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Executive Summary 



Cognitive interviews were conducted with teachers and their students to learn how to improve 
National Assessment of Education Progress (NAEP) questionnaire items asking about instructional 
practices and teacher professional development experiences. Items were purposively selected from those 
used in the 1996 and 1998 NAEP student and teacher background questionnaires in fourth grade reading, 
mathematics, and science, and eighth grade mathematics and science. NAEP items that were 
administered in analogous forms to teachers and their students were chosen, to facilitate validation. 
Additionally, items known or felt to be problematic were selected. 

Cognitive interviewing protocols were developed and administered to teachers in their 
classrooms. After the interviews were completed, their responses to instructional practice items were 
validated. Additional items were administered, to enable subsequent validation of their students’ 
responses. Then, cognitive interviewing protocols were administered to these teachers’ students. 
Students’ responses were compared to those provided by their teachers. When student and teacher 
responses were discrepant, further probing could be triggered. This ensured an understanding of the 
etiology of response discrepancies. 

Several general types of item problems were found. These are discussed below. 

Behavioral frequency items. Many NAEP questionnaire items were behavioral frequency items. 
These items ask how frequently a student engages in specific activities or a teacher employs specific 
instructional practices. Analyses of NAEP student behavioral frequency response data have not been 
especially informative. 

When teachers’ responses to the behavioral frequency items studied were compared to responses 
of their students, very high rates of discrepancy were observed. The average level of agreement between 
fourth grade students and their teachers on items that used a four-point rating scale was 38 percent; for 
eighth grade students and their teachers, the level of agreement was 5 1 percent. The level of agreement 
that would be expected by random guessing is 25 percent. 

There were several different reasons for these high rates of discrepancy. These include issues of 
item comprehension, scale problems, and other technical survey design issues. We feel the major source 
of these discrepancies is inherent in the task the students were being asked to perform. To answer these 
items, an individual can either count the number of times a behavior was performed or employ estimation 
strategies. When behaviors occur frequently and are of low salience, estimation strategies are typically 
employed. These estimates then have to be averaged over a time period (which is not always explicitly 
and unambiguously specified). The result of these calculations then must be compared with a series of 
response options and one of these options must be selected. Few eighth graders and fewer fourth graders 
are able to employ efficient and accurate estimation strategies and then average these estimates over a 
time period. In other words, the high rate of failure for these items is largely due to the fact that fourth 
and eighth grade students lack the cognitive skills and abilities required to synthesize an accurate 
response. 

Timeframe problems. The period of relevance for a survey item (e.g., the period over which 
behavioral frequencies are to be estimated) is called the item’s time frame. Our investigations enabled 
detection of the existence of implicit time frames that were inconsistent or in conflict with the item’s 
intended time frame. Items asking “When you study mathematics in school, ...” were almost always 
interpreted by teachers as asking about the current school year. Students would often think of a shorter, 
more recent period or a period of high salience. 
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Items that asked, “Have you ever done hands-on activities or projects in school with the 
following?” were interpreted by some, but not all students, as implicitly referring to the same time frame 
(current school year) asked about in the other items. Other students, particularly fourth graders, would 
interpret “ever” literally. They would include activities in other grades. 

Thus, vaguely defined explicit time frames and inconsistencies between an item’s explicit and 
implicit time frame contributed to response discrepancies. 

Problems with response options. Another time frame problem was associated with the response 
options for some of the frequency items inquiring about instructional practice. In our discussions with 
fourth grade teachers, we found that science lessons are not provided on a daily basis. Rather, they are 
episodic. However, the behavioral frequency scale for student science items was explicitly linked to the 
periods when science was being taught (“When you study science in school”). Accordingly, the response 
option, “Almost every day,” was explicitly interpreted as “Almost every day that science is taught.” 
Unfortunately, the next two response options (“Once or twice a week” and “Once or twice a month”) are 
implicitly linked to the calendar year. There is a conflict between the options’ implicit and explicit time 
frame. If science is usually taught twice a week and the focal behavior occurs nearly every time science 
is taught, the response alternatives “Almost every day” (explicitly interpreted) and “Once or twice a 
week” (implicitly interpreted) become indistinguishable. 

Comprehension problems. Certain words and phrases were not always interpreted as intended by 
the item writers, and, like linguistically complex items, were associated with comprehension problems. 
That is, these items were interpreted differently by different respondents. One commonly used phrase, 
“students in your class,” was discovered to be especially problematic for teachers. When teachers were 
asked “How often do the students in your class do ...?” they would either respond about the typical 
student or about how frequently any student in their class engaged in the focal behavior. Accordingly, a 
response of “almost every day” could mean that on most days at least one student used a computer or it 
could mean that the typical student used a computer nearly every day. 

Survey items often included parenthetical examples to clarify the meaning of a technical term or 
construct. In many cases, students were unable to generalize from the examples to the intended construct. 
In these cases, the examples become the focus of the question. For example, when fourth graders were 
asked about whether they had ever done any hands-on activities with “Chemicals (for example, mixing or 
dissolving sugar or salt in water)”, many would only focus on the examples. For these students, the item 
became a question about whether they had ever mixed or dissolved sugar or salt in water. 

Numerous comprehension issues, particularly those associated with the understanding of terms 
relating to specific instructional practices (e.g., “real-world problems,” “hands-on activities,” “science 
demonstration”) as well as other words and phrases are discussed in greater detail in sections 3and 4. 

Problems with list format items: Loss of context. Another problem was associated with the 
presentation of items in a list format. In list format items, an item stem will ask a question such as “When 
you do mathematics in school, how often do you do each of the following?” A rating scale will be 
provided, together with a list of several different behaviors. For example: 
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When you do mathematics in school, how often do you do each of the following? 




Fill in one box on each line. 




Almost 


Once or 


Once or 


Never or 




every 


twice a 


twice a 


hardly 




day 


week 


month 


ever 


1. Do mathematics problems from textbooks 


□ 


□ 


□ 


□ 


2. Do mathematics problems on worksheets 

3. Solve mathematics problems with a 


□ 


□ 


□ 


□ 


partner or in small groups 


□ 


□ 


□ 


□ 


9. Discuss solutions to mathematics 


problems with other students 


□ 


□ 


□ 


□ 


10. Use a computer 


□ 


□ 


□ 


□ 



As the respondent went through this list, it was not unusual to forget the conditions specified in 
the item stem. By the time the respondent got to “use a computer,” the respondent would not infrequently 
lose the item’s context and respond about general usage of a computer. The context of using a computer 
for doing mathematics in school had been lost. This loss of context occurred many times, for a variety of 
items, for both students and teachers. 

Problems with “Check all that apply ” items. Survey design and layout issues were the source of 
some problems. “Check all that apply” items provide a respondent with a list of behaviors, attributes, or 
characteristics and request the respondent to check all that apply. When a student did not understand 
what an item in the list meant, the student would usually (but not always) leave the box alongside this 
alternative unchecked. An unchecked box could either reflect the fact that the respondent did not 
understand the item, refused to answer the item, or that the item did not apply to the respondent. 

Other problems and issues. Issues associated with the development of items relevant to each 
different subject area are discussed in the report. For example, elementary grade science items for 
students must take into account the episodic nature of science lessons, the integration of science lessons 
with other lessons, and difficulties inherent in identification of a lesson as a science lesson (e.g., Should a 
discussion of a recent earthquake be considered “current events” or “science?”). Similarly, the 
development and analysis of items dealing with reading and mathematics instruction should take into 
account the fact that instruction in these areas may be integrated with instruction in other subject areas. 

Overall summary. Cognitive interviews, with validation components, enabled detection of many 
survey design problems and an understanding of the reasons for their occurrence. As a result, these 
problems can be avoided in the development of future NAEP and other background survey items. 
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1. INTRODUCTION 



NAEP Background Questionnaires 

The National Assessment of Educational Progress (NAEP) is involved in a major effort to 
measure the educational achievement of our nation’s children. As part of this effort, when students take 
NAEP achievement tests, they also answer questions about a variety of background factors that are 
known or believed to be related to performance on these tests. Some of the questions are related to home 
background factors (e.g., parent education, TV watching, and homework). Other questions ask both 
students and teachers about instructional practices such as the use of computers in mathematics or the 
frequency of group activities in reading. Students are asked attitudinal questions about specific subjects; 
teachers are asked background items about professional development. 

There has been concern about the quality of student background data. Data from fourth graders 
on specific factors (e.g., parent education) have been deemed unreliable due to high omission rates. For 
similar reasons, data from eighth grade students, in particular from minority students, have also been of 
questionable reliability. Accordingly, a study was undertaken in 1996-97 to investigate the quality of the 
fourth and eighth grade home background items. A cognitive interviewing protocol was developed, 
which included a validation component. Parents of the students were interviewed along with their 
children, providing information to validate their child’s responses. These procedures enabled 
identification of item problems that would not otherwise have been detected (Levine et al., 2001). 

This study is an extension of the previous effort. Cognitive interviewing protocols were again 
developed; these included validation components. However, this time the questions were primarily 
concerned with instructional practices. So, students’ teachers provided the validating information. 

Instructional background items from the 1996 and 199S NAEP background questionnaires in 
fourth grade reading, mathematics, and science, and eighth grade mathematics and science were selected 
for investigation. A total of 66 students and 12 teachers participated in this study. 



Cognitive Interviewing and Validation 

Cognitive interviewing (also known as verbal reporting [Willis et al., 1991]) of questionnaire 
respondents is a form of interview used to uncover the mental processes involved when a respondent 
reads and responds to survey questions (Willis et al., 1999). Cognitive interviews are effective in 
determining how respondents comprehend survey items and what strategies they use to devise answers. 
Such interviews are primarily conducted in the course of pretesting surveys, to identify sources of 
respondent confusion and misunderstanding (Krosnick, 1999; Fowler and Canned, 1996; Schaeffer and 
Maynard, 1996). They may lead to the verification of an expected question problem or the discovery of a 
problem that was unanticipated (Willis et al., 1999; DeMaio and Rothgeb, 1996). Cognitive interviewing 
can facilitate not only finding a problem but also fixing that problem (Willis et al., 1999). 

In most cognitive interviews, two basic techniques are utilized: “think-alouds” and verbal probing 
(Willis et al., 1999). In the think-aloud technique, the interviewer asks the subject to report what he or 
she is thinking as he or she is answering an item. Often a simple question such as “Can you tell me what 
you’re thinking?” is used. The interviewer records the process the respondent describes as he or she 
arrives at an answer. In addition, specific probes related to each item (e.g., requests to paraphrase the 
question or requests to define words and phrases) are developed and then administered after the 
participant has produced a response to the survey question. 
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The use of validating information can be of tremendous benefit to the cognitive interviewer 
(Huberman and Levine, 1999). Although think- alouds and normal probing enable the identification of 
many item problems, there are occasions when an individual’s think-aloud and responses to probes will 
fail to reveal the existence of an incorrect response. But, if the interviewer knows the response is 
problematic (as a result of validating information), further probing can be employed until reasons for the 
otherwise undetectable item problem are determined. Validation data also lend themselves to simple, 
tabular presentations of the effectiveness of an item. 

Validation information can be quite difficult to obtain. Not infrequently, the item being validated 
is a self-report item where only the respondent knows “truth.” Effective procedures for validating self- 
report data employing focused retrieval, extensive retrieval, varied retrieval, and attempts to evoke 
multiple representations of the construct of interest through the reconstruction of a calendar/diary for a 
time period of interest have been used to validate self-reports of hours worked (Edwards, Levine, and 
Cohany, 1989). In a similar fashion, to improve the quality of eyewitness reports of a crime, these 
procedures have been employed in a technique called (coincidentally) “cognitive interviewing.” The 
principles underlying this type of cognitive interviewing are (Fisher and Quigley, 1992): 

1) Context reinstatement . The same psychological environment in which the event occurred is 
created to increase the validity of recall (Tulving and Thomson, 1973). 

2) Focused retrieval. The interviewer gets the respondent to expend effort and engage in 
uninterrupted concentration (Kahneman, 1973). 

3) Extensive retrieval . The more retrieval efforts the respondent makes, the more successful 
recall will be. Thus, the respondent is encouraged to search through memory even if she 
thinks that she has recalled everything (Roediger and Payne, 1982). 

4) Varied retrieval. The use of different retrieval probes is more effective than the use of a 
single retrieval probe (Anderson and Pichert, 1978). 

5) Multiple representations . The construct of interest will have different mental representations 
in the respondent’s memory. Each different representation is evoked and probed separately 
(Fisher and Chandler, 1984). 

These procedures have been shown to be effective in enhancing dietary recall (Fisher and 
Quigley, 1992). 

Teachers were used as a source of validation data for student responses to items dealing with 
instructional practices and classroom behaviors. However, before the teacher data were used to validate 
student responses, these teacher data underwent validation. A calendar exercise incorporating the above 
techniques was used for this purpose. Thus, student data were compared to validated teacher data. 



Overview of the Report 

We describe the procedures used to choose items for investigation and to collect and analyze data 
in section 2. We present results and general findings in section 3 and provide detailed findings for each 
item investigated in section 4. In section 4, which is organized by grade and subject, we also discuss 
findings and provide recommendations for item modifications to correct specific item problems. 
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2. METHODOLOGY 



Survey Development (Item Selection) 



Overview 



Since resources did not permit all NAEP items to be investigated, three different sets of criteria 
were used to select items for cognitive laboratory investigations: 

1) Fourth grade mathematics background items that were deemed problematic based on 
secondary analyses of NAEP data were selected. 

2) NAEP student items that could be validated through administration of parallel NAEP teacher 
items were selected. 

3) Items that were of special programmatic interest or were otherwise felt to be problematic 
were selected. 

These items were not intended to be a random sample of all NAEP items. Rather, the purpose of 
these investigations was to enable the development of higher quality student and teacher survey items in 
the future, through investigation of items known or felt to be problematic or of programmatic interest. 



Procedures 

The fourth grade 1996 NAEP student mathematics questionnaire contained items asking about the 
frequency with which mathematics related activities occurred and the frequency with which calculators 
were used for classwork, homework, and tests or quizzes. Analogous items were also asked of teachers. 
Secondary analyses were conducted on these items. 

If these mathematics items were eliciting valid responses, there should be evidence of reliability 
or agreement among the responses of survey respondents (i.e., consensual validation). That is, responses 
to these items should be relatively constant among the students in a class. The presence of substantial 
within-class variation is an indicator of potential item problems. In addition, when similar items were 
administered to both students and teachers, student responses should be the same as their teacher’s 
responses. 

Student responses were compared with their teacher’s responses to identify those items with the 
greatest mean differences. 1 For all cases in which two or more student responses could be linked with a 
teacher, variation within a classroom was calculated in four different ways: 

1) The absolute value of the deviation of student responses from their class’s mean value was 
determined, and divided by the number of student respondents in the class, to create a “mean 
deviation” score. 

2) Each classroom’s standard deviation was calculated and averaged over all classrooms to 
create a “mean standard deviation.” 



1 Behavioral frequency responses were treated as interval data “Almost every day” was assigned a value of 4; “Once or twice a 
week,’’ a value of 3; “Once or twice a month,’’ a value of 2; and “Never or hardly,” a value of 1 . 
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3) The absolute value of the deviation of student responses from their teacher’s response was 
determined for each classroom and averaged over all classrooms, to create a “mean deviation 
from the teacher’s report.” 

4) The deviation of student responses from their teacher’s response was averaged over all 
classrooms to estimate a “Student - Teacher deviation.” 

In ranking items according to the amount of student response variation within a class, within class 
mean deviations and within class standard deviations yielded nearly identical orderings of the items. (See 
table 1.) Two of the four items with the greatest within class variation were concerned with calculator 
usage: “Using a calculator for mathematics homework” and “Using a calculator for mathematics class 
work.” These variations may simply reflect the natural variation of these behaviors among students in a 
class. The other two of the four most variable items (“Talking to the class about your mathematics work” 
and “Discussing solutions to mathematics problems with other students”) may similarly represent 
teaching practices for which variation among students is to be expected. Certain students may be called 
upon more frequently than others to talk about their work or may discuss solutions with other students 
more frequently than other students. Comparisons of student responses with teacher responses for these 
items revealed the greatest variation. 
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Table 1. Within Classroom Variations in Student Responses to Mathematics Behavior Items 



When you do mathematics in school, 
how often do you do each of the 
following? 


Mean 

Dev. 


Mean 

S.D. 


Student 

Mean 


S.D. 


# 

Students 


# 

Classes 


14. (Use a calculator for math) homework 


0.97 


1.17 


2.33 


1.29 


6,369 


556 


7. Talk to class about your mathematics 
work 


0.94 


1.15 


2.09 


1.23 


6,297 


557 


13. (Use a calculator for math) class work 


0.92 


1.13 


2.42 


1.27 


6,402 


557 


9. Discuss solutions to mathematics 
problems with other students 


0.86 


1.06 


2.33 


1.14 


6,486 


557 


5. Write a few sentences about how you 
solved a mathematics problem 


0.83 


1.03 


2.32 


1.16 


6,449 


557 


10. Use a computer 


0.83 


1.02 


1.89 


1.14 


6,472 


I 557 


8. Do 10 or more practice problems in 
mathematics by yourself 


0.79 


1.00 


3.14 


1.09 


6,466 


557 


4. Work with objects like rulers, 

counting blocks, or geometric shapes 


0.75 


0.94 


2.18 


1.08 


6,488 


557 


3. Solve mathematics problems with a 
partner or in a small group 


0.73 


0.91 


2.03 


1.03 


6,487 


556 


11. Use a calculator 


0.70 


0.88 


1.97 


1.01 


6,476 


“557 


15. (Use a calculator) for tests or quizzes 


0.65 


0.81 


1.65 


1.07 


6,522 


557 


1. Do mathematics problems from 
textbooks 


0.64 


0.83 


3.26 


0.92 


6,335 


557 


2. Do mathematics problems on work 
sheets 


0.62 


0.79 


3.15 


0.97 


6,509 


557 


6. Take mathematics tests 


0.52 


0.68 


2.39 


0.79 


6,497 


557 



NOTE: The stem for items 13-15 was “For mathematics, how often do you use a calculator for each of the 
following activities?” 



Teachers generally reported a greater frequency of occurrence of these behaviors than did 
students. (See table 2.) It was subsequently noted that the question, “How often do your students do each 
of the following?” can be interpreted in two different ways: 

1) How often does the TYPICAL student in your class perform the focal behavior? 

2) How often does ANY student in your class perform the focal behavior? 

Naturally, the second interpretation will lead to higher behavioral frequencies. This may explain 
why, for most items, and particularly for those for which a wide range of between student variation was 
expected, teachers tended to report greater behavioral frequencies than their students. These patterns of 
variation suggest that these items are not functioning as intended. 
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Table 2. Comparisons of Student and Teacher Responses to Mathematics Behavior Items 



When you do mathematics in school, 
how often do you do each of the 
following? 


Absolute Mean 
Deviation 


Overal 


Mean 


Student- 

Teacher 

Deviation 


Student 


Teacher 


Student 


Teacher 


7. Talk to class about your mathematics 
work 


0.94 


1.44 


2.09 


2.93 


-0.84 


9. Discuss solutions to mathematics 
problems with other students 


0.86 


1.20 


2.33 


3.00 


-0.67 


5. Write a few sentences about how you 
solved a mathematics problem 


0.83 


1.05 


2.32 


2.10 


0.23 


10. Use a computer 


0.83 


1.16 


1.89 


2.49 


-0.59 


3. Solve mathematics problems with a 
partner or in a small group 


0.73 


1.13 


2.03 


2.90 


-0.86 


11. Use a calculator 


0.70 


0.82 


1.97 


2.03 


-0.06 


1 . Do mathematics problems from 
textbooks 


0.64 


0.68 


3.26 


3.46 


-0.20 


2. Do mathematics problems on work 
sheets 


0.62 


0.79 


3.15 


2.96 


0.18 


6. Take mathematics tests 


0.52 


0.58 


2.39 


2.32 


0.07 



NOTE: The item wording is from the Student questionnaire. Only directly comparable items are presented in this 



table. 



Nonetheless, many of the other behavioral frequency items were items for which between student 
variation was expected to be minimal. And, since these other items referred to class-wide practices, 
agreement with teacher responses would be predicted to be high — if the items were functioning as 
intended. The item with the fifth greatest amount of within class variation — “Write a few sentences about 
how you solved a mathematics problem” — represents a teaching practice that all students in a given 
classroom should be expected to engage in with roughly equal frequencies. The substantial within class 
variation suggests possible item problems. Student reports were also discrepant from their teacher’s 
reports — by an average of almost one scale unit. For these reasons, this item was chosen for 
investigation. 

In addition, items that could be validated through administration of ancillary questions to teachers 
and items of programmatic interest or those that were thought to be problematic were selected. Selection 
of items for the other fourth grade subjects and eighth grade mathematics and science were based on these 
criteria. No other secondary analyses were conducted. 

Selected items were used to create fourth grade student surveys in mathematics, science, and 
reading, and eighth grade student surveys in mathematics and science. Because fourth grade teachers 
usually teach all subjects, only one teacher survey covering all three subject areas was created. Separate 
surveys in mathematics and science were developed at the eighth grade level because of the 
departmentalized nature of instruction in most middle schools. Each survey requested information on the 
frequency of instructional practices such as taking tests, using calculators and computers, classroom 
presentations, field trips, and homework related to each subject area. In addition, the teacher surveys 
asked questions about the teacher’s professional development activities in the past year. 

Protocols for each survey were developed. The protocols provided a variety of optional probes 
(e.g., word definition and paraphrasing), to be used when deemed appropriate. The protocols were 
reviewed both internally and by Dr. Robert Belli, a cognitive survey researcher at the University of 
Michigan’s Institute for Survey Research. In addition, interviewers were trained to develop and employ 
probes, as necessary, during the interviews. 
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Protocol Refinement 



Protocols underwent revisions during the data collection process. These revisions were based on 
preliminary findings and were intended to increase the utility of the data collected. There were three 
major types of revisions. 

We noticed that individuals responding to items presented in list format were losing context. 

That is, they would respond to the component of the item that was in a list rather than to the question 
posed in the item’s stem. For example, the stem of an item might ask, “When you teach mathematics , 
how often do you do each of the following?” Several items would then be listed, including an option such 
as, “ Use computers. ” If a respondent answered about his or her use of computers for purposes other than 
teaching mathematics, he or she is said to have lost context. The context of performing the behavior for 
the purpose of teaching mathematics that appears in the stem is no longer salient at the time of response. 

Since our protocol required the respondent to think aloud while responding to an item, it could be 
argued that this loss of context is artifactual. The process of thinking aloud, focusing on a specific item, 
might be responsible for the respondent forgetting the item’s stem. So, procedures were modified for the 
examination of items in list format: The respondent was asked to answer all of the listed items, without 
thinking out loud. After the last item in the list, (s)he was asked to describe what was going on in his or 
her mind when (s)he was answering the question. In other words, instead of using concurrent think 
alouds, retrospective think alouds were employed for the investigation of these items. Regardless of 
whether concurrent or retrospective think alouds were used, respondents continue to lose context, 
demonstrating that the loss of context was not artifactual. 

The second type of modification involved alteration of item wordings. For certain items, there 
were clear comprehension problems. For example, no fourth grader knew what was meant by “e.g.” 
Rather than demonstrate this over and over, decisions were made to employ alternative item wordings 
(such as the replacement of “e.g.” with “for example”) to allow determination of the effectiveness of the 
alternative wording. Similarly, the term “novels” was changed to “books with chapters” and “undecided” 
was changed to “not sure.” 

The third type of modification involved alteration of behavioral frequency scales. Behavioral 
frequency items were typically associated with very high levels of non-agreement. Rather than 
continuing to use a four-point frequency scale, a three-point scale was developed and employed. The 
reduction of the number of points in a rating scale will increase the amount of agreement, if for no other 
reason than an increase in the number of “chance” agreements. There was little evidence that this 
procedure substantially increased agreement beyond what might be expected by chance. 

Participant Recruitment and Selection 

In order to get a heterogeneous group of participants, three San Francisco Bay Area school 
districts were contacted. Informational materials and consent forms were prepared and sent to teachers in 
six different schools in these districts. As an incentive for participation, teachers were offered $100 and 
students $50 for their time. After teachers agreed to participate in the study, they were asked to distribute 
informational materials and consent forms to their students. Students and their parents were encouraged 
to either call the American Institutes for Research directly to set up an appointment for an interview or to 
return the signed consent form to the teacher. Potential participants were screened to allow selection of a 
diverse sample with respect to the household’s annual income level and the race/ethnicity of the student. 
Each teacher was able to help us recruit between three and eight of his or her students. 

Spanish versions of the informational materials were also prepared and distributed to students. 
Staff fluent in Spanish worked with Spanish-speaking parents. However, children needed to be able to 
speak and read English in order to participate in the study, since this was a requirement to participate in 
the NAEP assessment. 
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A total of 12 teachers (six fourth grade teachers and six eighth grade teachers) and 66 of their 
students (35 fourth graders and 31 eighth graders), representing five elementary and middle schools were 
interviewed for the study. One-third (33 percent) of the student sample was from low-income households 
(below an annual income of $30,000) and about two-fifths (41 percent) of the sample were minority 
students (mostly of Hispanic origin). 



Interview Procedures 



Teacher Interviews 

The teacher interviews lasted about two hours and consisted of two phases. In the first phase, the 
survey items were administered. The teacher was asked to read the questions aloud and was reminded to 
think aloud to provide insights into the cognitive processes that he or she employed in responding to the 
items. In addition, specific probes and paraphrasing requests were used to obtain additional information 
about the response process. Typical probes included: “What do you think they mean by [technical 
term]?” and “What do you think this question is asking?” 

The second phase of the interview was designed to validate some of the teachers’ responses in 
Phase 1. For cognitive reinstatement purposes, interviews were conducted in the teacher’s classroom. 

The teacher was asked to reconstruct the past week’s activities in class through a calendar exercise. To 
this end, weekly matrices were designed to help the teacher recall the frequency of certain instructional 
practices occurring each day. The teacher recalled each day of the week by first thinking of important or 
atypical events that occurred during the past week (e.g., staff meetings, sick students, and special events). 
These were written onto the matrix to serve as cognitive anchors and facilitate recall of each day’s events. 
Then, the teacher was asked about the day’s lesson in the subject area of interest (e.g., what the teacher 
taught that day, whether he or she used any special materials such as manipulatives, and whether the 
teacher utilized technology). To evoke multiple representations, the teacher was also asked to think about 
a specific student and what this student did during the lesson in question. All of these extensive retrieval 
activities were intended to facilitate recall. After the interviewer felt this effort resulted in retrieval of a 
clear representation of the day’s lesson, the teacher was then asked to report which of several instructional 
practices occurred on that day in the subject area of interest. This process was repeated for each subject 
and each day. Responses were documented on the matrix, as well. Then, the number of days on which 
each behavior of interest was performed was tallied, producing frequencies for each behavior for the past 
week. Respondents were then asked whether the past week was typical with respect to instructional 
practices. If not, they were requested to make adjustments to account for the atypicality of the week, and 
these frequencies were then compared to the frequencies given in the first phase of the interview. 
Discrepancies were probed, and the most accurate answer (as determined by the teacher) was noted. 

Fourth grade teachers’ responses to the following behavioral frequency items were validated 
through use of a calendar/diary exercise: 

READING LESSONS: TEACHER’S BEHAVIORS 

♦ Ask students to talk with each other about what they have read 

♦ Ask students to write about something that they have read 

♦ Ask students to do a group activity or project about what they have read 

♦ Help students understand new words 

♦ Ask students to make predictions about what they read as they are reading it 

♦ Ask students to make generalizations and draw inferences based on what they have read 
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SCIENCE LESSONS: STUDENT’S BEHAVIORS 

♦ Discuss science in the news 

♦ Do hands-on activities or investigations in science 

♦ Talk about measurements and results from student’s hands-on activities 

♦ Use library resources for science 

♦ Use computers for science 

SCIENCE LESSONS: TEACHER’S BEHAVIORS 

♦ Talk to the class about science 

♦ Do a science demonstration 

♦ Show a science video or science television program 

♦ Use computers for science (e.g., science software, telecommunications) 

MATHEMATICS LESSONS: STUDENT’S BEHAVIORS 

♦ Do mathematics problems from textbooks 

♦ Do 10 or more practice problems by themselves 

♦ Do mathematics problems on worksheets 

♦ Solve mathematics problems in small groups or with a partner 

♦ Work with objects like rulers 

♦ Work with counting blocks or geometric shapes 

♦ Use a calculator 

♦ Take mathematics tests 

♦ Write a few sentences about how to solve a mathematics problem 

♦ Talk to the class about their mathematics work 

♦ Discuss solutions to mathematics problems with other students 

♦ Use a computer 

Eighth grade science and eighth grade mathematics teachers’ responses to the following 
behavioral frequency items were validated through use of a calendar/diary exercise: 

SCIENCE LESSONS: TEACHER’S BEHAVIORS 

♦ Talk to the class about science 

♦ Do a science demonstration 

♦ Use computers for science 

SCIENCE LESSONS: STUDENT’S BEHAVIORS 

♦ Discuss science in the news 

♦ Work with other students on a science activity or project 

♦ Give an oral science report 
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♦ Do hands-on activities or investigations in science 

♦ Talk about measurements and results from their hands-on activities or investigations 
MATHEMATICS LESSONS: STUDENT’S BEHAVIORS 

♦ Do mathematics problems from textbooks 

♦ Do mathematics problems on worksheets 

♦ Solve mathematics problems in small groups or with a partner 

♦ Work with objects like rulers 

♦ Work with counting blocks or geometric shapes 

♦ Take mathematics tests 

♦ Write a few sentences about how to solve a mathematics problem 

♦ Talk to the class about their mathematics work 

♦ Write reports or do mathematics projects 

♦ Discuss solutions to mathematics problems with other students 

♦ Work and discuss mathematics problems that reflect real-life situations 

♦ Use a computer 

Our major motivation for undertaking this validation was to produce the highest quality criterion 
measures against which student responses could be compared. Major adjustments were the exception 
rather than the rule; teachers’ reconstructed responses were usually the same as their initial responses. 

Since the student surveys contained some items that were not included in the teacher survey, the 
teachers were asked additional questions that could be used to validate the students’ answers. For 
example, the teachers were asked about homework, science field trips, the use of calculators in 
mathematics, and the use of assessments in mathematics, reading, and science. 

Following each teacher interview, we prepared a summary of the interview focusing on item 
problems. 



Student Interviews 

Before the student interviews were conducted, teacher responses and any other relevant 
information from the teacher interviews were recorded on the student protocols to facilitate the 
identification of discrepancies and the triggering of probes. 

The three fourth grade subject area surveys consisted of between 12 to 19 items. Since three 
subject area surveys (with their associated protocols) could not be administered in a two-hour cognitive 
interview session, each fourth grader was administered two of three surveys — mathematics and science, 
mathematics and reading, or science and reading. Because of the departmentalized nature of middle 
schools, as mentioned earlier, eighth graders answered either a 23-item mathematics survey or a science 
survey with 23 items. 

Similar to the first phase of the teacher interviews, the students were asked to read the questions 
aloud. This facilitated detection of potential language and comprehension problems. For example, when 
a student could not read or pronounce a word, it could indicate a comprehension problem. In these cases, 
the interviewer would make sure to probe the student’s understanding of the particular word. 
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The students were continually encouraged to think aloud and, like the teacher interviews, specific 
probing and paraphrasing techniques were used to obtain additional information about the student item 
response process. When a student’s response differed from that of his or her teacher, the interviewer tried 
to determine the reason for the discrepancy by administering additional probes about the item (e.g., 
asking for further clarification about how estimates were produced or verifying comprehension of the 
item). For instance, an eighth grade science student indicated that he had not been on a science field trip 
this year, even though the interviewer knew from his teacher’s response that the science class had been on 
a field trip to the NASA Ames Research Laboratories. With this information in mind, the interviewer 
asked if the student had been on ANY field trips this year and the student responded that he had been on a 
field trip to the NASA Ames Research Laboratories! However, the student did not consider this a science 
field trip. An example of a science field trip in his mind would be to visit a research institute like the 
American Institutes for Research. Without the teacher’s response to validate the student’s answer, this 
item problem (the definition of “a science field trip”) might not have been detected. 

After each student interview, a summary was prepared focusing on item problems and the reasons 
for any discrepancies between the student’s and his or her teacher’s answers. 

Analysis 

To summarize results, we compared students’ responses to their teachers’ adjusted responses. 
Discrepancy rates were calculated across all student-teacher item pairs. 

When discrepancies occurred, the student and teacher summaries were analyzed to identify the 
reasons for the discrepancies. In nearly all cases, the teacher’s validated response was considered to be 
the correct response. However, situations would occasionally arise which strongly suggested that the 
teacher had misinterpreted the question. These reasons are presented in item-by-item discussions and are 
used to inform suggested actions. 

Teachers’ responses to the selected behavioral frequency items that were validated through the 
calendar exercise were compared with their survey (Phase I) responses. The frequencies and magnitudes 
of these discrepancies are summarized in the next section. 
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3. RESULTS AND DISCUSSION 



Teacher Self-reported Behavioral Frequency Data 

The final estimates produced by teachers after the validation exercise were compared to their 
responses to the original presentation of these items. These items were rated on a four-point frequency 
scale: 



1) Never or hardly ever 

2) 1-2 times per month 

3) 1-2 times per week 

4) Almost every day 

The proportion of responses that were changed as a result of the enhanced recall procedures and 
the proportion of responses that changed by two or more scale points (defined as a major discrepancy) are 
summarized in Tables 3 and 4. 



Table 3. Discrepancy Rates of Fourth Grade Teachers to Selected Behavioral 
Frequency Items, by Subject Area 



Subject Area 


Discrepancy Rate 


Any Discrepancy 


Major Discrepancy 


Science 


37% 


2% 


Math 


36% 


11% 


Reading 


36% 


3% 



NOTE: Results are based on the responses of six teachers. 



Table 4. Discrepancy Rates of Eighth Grade Science and Mathematics Teachers to 
Selected Behavioral Frequency Items, by Subject Area 



Subject Area 


Discrepancy Rate 


Any Discrepancy 


Major Discrepancy 


Science 


12% 


0% 


Math 


25% 


8% 



NOTE: Results are based on the responses of three science and three mathematics teachers. 



Major discrepancies were relatively rare. They generally resulted from the teacher’s 
misinterpretation of the item’s intent. Minor discrepancies reflected both the difficulty of the task 
(producing behavioral frequency estimates) and gaps between the scale points. Gaps occurred between 
all of the scale points. Behaviors performed about three times per week could either be classified as 
“Almost every day” or “1-2 times per week;” behaviors performed approximately three times a month 
could be classified as “1-2 times per week” or “1-2 times per month;” behaviors that occurred between 
three and six times per year could be classified as “Never or hardly ever” or “1-2 times per month.” 

Minor changes in a behavioral frequency estimate would result in its classification in an adjacent response 
category. 
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Comparison of Student and Teacher Behavioral Frequency Data 



From the think-alouds and the validation data provided by teachers, it was possible to compare 
students’ answers with their teacher’s responses, identify inconsistencies, and determine the reasons that 
these inconsistencies occurred. Discrepancy rates for fourth graders are presented in tables 5-7. In 
these tables, the 4-point scale refers to the same behavioral frequency scale used in the teacher 
questionnaire: 

1 ) Never or hardly ever 

2) 1-2 times per month 

3) 1-2 times per week 

4) Almost every day 

As noted in the previous section, very high student-teacher response inconsistency rates were 
associated with use of this scale. Accordingly, a simpler, 3-point scale was developed and administered 
to students. In this scale, the first response option was modified and the middle response options were 
combined: 

1) Never 

2) Sometimes 

3) Almost every day 

This alternative scale was implemented for heuristic purposes. In a qualitative study of this kind, with 
small numbers of respondents, our intention was not to enable the measurement and detection of 
statistically significant differences. Rather, our intent was to roughly gauge the impact of an alternative 
scale. 
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Table 5. Discrepancy Rate for Selected Fourth Grade Science Items, by Type of Scale 



When you study science in school, how often do you do each of 
the following: 


4-point 

scale 

(n=16) 


3-point 

scale 

(n=7) 


Discuss science in the news 


69% 


43% 


Do hands-on activities in science 


75% 


0% 


Talk about measurements and results from your hands-on activities 


75% 


29% 


Use a computer for science 


13% 


57% 


Use library resources for science 


69% 


29% 


When you study science in school, how often does your teacher 
do each of the following? 






Talk to the class about science 


81% 


29% 


Do a science demonstration 


88% 


29% 


Show a science videotape or science television program 


63% 


71% 


Use computers for science (e.g., such as science software, 
telecommunications) 


13% 


33% 



NOTE: The number of students (n) is the modal number of respondents to each item. 



Table 6. Discrepancy Rate for Selected Fourth Grade Mathematics Items, by Type of Scale 



When you do mathematics in school, how often do you do each 
of the following: 


4-point 

scale 

(n=19) 


3-point 

scale 

(n=6) 


Do mathematics problems from textbooks 


68% 


33% 


Do mathematics problems on worksheets 


74% 


17% 


Solve mathematics problems with a partner or in small groups 


74% 


100% 


Work with objects like rulers, counting blocks, or geometric 
shapes 


74% 


67% 


Write a few sentences about how you solved a mathematics 
problem 


61% 


67% 


Take mathematics tests 


32% 


33% 


Talk to the class about your mathematics work 


89% 


100% 


Do 10 or more practice problems in mathematics by yourself 


79% 


50% 


Discuss solutions to mathematics problems with other students 


58% 


100% 


Use a computer 


37% 


0% 


Use a calculator 


42% 


66% 


This year in school, how often have you taken mathematics tests 
where you were asked to provide detailed solutions to problems 
you had not worked on before? 


88% 


60% 



NOTE: The number of students (n) is the modal number of respondents to each item. 
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Table 7. Discrepancy Rate for Selected Fourth Grade Reading Items, by Type of Scale 



When you have reading assignments in school, how often does 
your teacher do each of the following: 


4-point 
scale 
( n =1 6) 


3-point 

scale 

(n=7) 


Ask you to do a group activity or project about what you have read 


50% 


57% 


Ask you to talk to other students about what you have read 


79% 


29% 


As you to write about something you have read 


33% 


71% 


Help you break words into parts 


62% 


50% 


Help you understand new words 


60% 


86% 


This year in school, how often have you been asked to write long 
answers to questions on tests or assignments that involved reading? 


67% 


83% 



NOTE: The number of students (n) is the modal number of respondents to each item. 



The average discrepancy rate for a 4-point scale item was 62 percent; for a 3-point item, 51 
percent. Alternatively, these rates can be presented as their complement: agreement rates. The average 
agreement rate for a 4-point scale item was 38 percent; for a 3-point item, 49 percent. Through totally 
random guessing, agreement rates for 4-point scale items of 25 percent and for 3-point scale items of 33 
percent would be expected. There is little evidence that decreasing the number of scale points 
substantially increased the levels of student-teacher agreement, relative to what would be expected by 
chance. This suggests that the reasons for the high discrepancy rates are associated with factors other 
than the scale employed. 

Discrepancy rates for eighth grade science and mathematics items are presented in tables 8 and 9. 
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Table 8. Discrepancy Rate for Selected Eighth Grade Science Items 



When you study science in school, how often do you do each of the 
following: 


(n=11) 


Discuss science in the news 


18% 


Work with other students on a science activity or project 


82% 


Give an oral science report 


55% 


Do hands-on activities or investigations in science 


73% 


Talk about the measurements and results from your hands-on activities or 
investigations 


82% 


Go outside to observe or measure things 


36% 


Design and carry out your own science investigations 


55% 


When you study science in school, how often does your teacher do each of 
the following: 




Talk to the class about science 


27% 


Do a science demonstration 


45% 


Use computers for science (e.g., science software, telecommunications) 


18% 


About how often does your science class go on a science field trip? 


18% 


About how often does a guest speaker come to speak to your science class? 


9% 


Which best describes the science course you are taking? 


67% 


About how often do you study science in school? 


36% 


Do either you or your teacher save your science work in a portfolio? 


55% 


Do you ever do science projects in school that take a week or more? 


27% 



NOTE: The number of students (n) is the modal number of respondents to each item. 



Table 9. Discrepancy Rate for Selected Eighth Grade Mathematics Items 



When you do mathematics in school, how often do you do each of the 
following: 


(n=20) 


Do mathematics problems from textbooks 


65% 


Do mathematics problems on worksheets 


60% 


Solve mathematics problems with a partner or in small groups 


55% 


Work with measuring instruments or geometric solids 


55% 


Write a few sentences about how you solved a mathematics problem 


50% 


Take mathematics tests 


25% 


Talk to the class about your mathematics work 


80% 


Do 10 or more practice problems by yourself 


38% 


Discuss solutions to mathematics problems with other students 


60% 


Use a computer 


45% 


Use a calculator 


20% 


Write reports or do mathematics projects 


20% 


Work and discuss mathematics problems that reflect real-life situations 


65% 


Do either you or your teacher have a portfolio with your mathematics work in it? 


30% 


What kind of mathematics class are you taking this year? 


23% 



NOTE: The number of students (n) is the modal number of respondents to each item. 



The average discrepancy rate for 4-point scale items administered to eighth graders was 49 
percent. The agreement rate, the complement of this percentage, was 51 percent. This agreement rate 
was greater than that of fourth graders (38 percent) for comparable items. It should be noted that there are 
legitimate reasons to expect high discrepancy rates for some of these behaviors. Some students may be 
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called upon to “Talk to the class about your mathematics work” much more frequently than others. 
However, many of these behaviors are ones for which the within classroom variance would be expected 
to be minimal. Taking tests is one such example; doing problems from textbooks or worksheets are other 
examples. 

The reasons for the high discrepancy rates for these and other types of items are summarized 

below. 

General Findings 

Behavioral Frequency Items 

The major reasons for the high discrepancy rates associated with behavioral frequency items 

were: 

1) Many of the behaviors of concern were not being interpreted as intended by the item writers. 2 

2) Many individuals (especially students) lack the cognitive abilities to synthesize a behavioral 
frequency accurately, particularly when: 

♦ the behavior does not occur on a regular basis 

♦ the behavior occurs frequently, 

♦ the behavior is of low salience, and 

♦ the time period (i.e., the denominator for rate calculations) is either ambiguous, 
unspecified, or long. 

Behavioral frequency items have been extensively studied by survey researchers (Sudman, 
Bradbum, and Schwarz, 1995). Respondents usually estimate rare behaviors or events by counting them. 
Common events or behaviors, which are the focus of many of the behavioral frequency questions, are too 
indistinct for counting. Instead, estimation strategies have to be employed. The specific estimation 
strategies employed are dependent on the cognitive skills, abilities, and motivation of the respondent. 
There is little evidence that the fourth and eighth graders studied possess the skills and abilities required 
to accurately estimate frequencies for common, low salience behaviors. We were able to demonstrate that 
even when students understood what sorts of behaviors they were being asked about and could recall 
specific instances of their occurrence, they were often unable to estimate their frequency of occurrence 
accurately. Guessing was a common strategy. 

When a respondent cannot accurately estimate the numerator for a behavioral frequency, and 
when the denominator for the behavioral frequency is based on a time period that is vague or ambiguous, 
high discrepancy rates will be produced. Although many of the items were phrased in the current tense, 
many of the behaviors of interest do not occur at the same rate throughout the school year. Some days or 
weeks might be devoted to special projects or to preparation for standardized tests. Furthermore, certain 
behaviors are likely to be associated with specific units. Students might work with rulers quite a bit for 
some units and not at all for others. As one teacher pointed out, “What are my students going to do? 
They’re not going to average things out over the year.” In other words, even when respondents realize the 
item is asking about current behaviors, teachers will usually try to answer about the entire school year, 
while students will often respond about a shorter, more recent period or about a period of high salience. 

There were other issues that further contributed to these discrepancy rates. These included 
additional time frame issues, item comprehension problem, loss of context, and other factors. 



2 Item comprehension issues are discussed in greater detail in the next section. 
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Time Frame Issues 



The time period of interest was not always interpreted as intended by the item writers. For 
example, many behavioral frequency items begin with phrases such as: 

♦ “When you study science in school” 

♦ “When you have reading assignments in school” 

♦ “When you do mathematics in school” 

These items have implicitly bounded time frames. That is, the intent of these items is to provide 
an indicator of how often this behavior is performed over the school year. Most respondents realized that 
these behavioral frequency items were asking about current behaviors. However, there is no explicit 
linkage to the school year in these items. In fact, the specific period of relevance is quite vague and lends 
itself to many different reasonable interpretations. 

Time frame issues were exacerbated by problems with the associated scale when the denominator 
for the frequency calculations was an event that did not occur daily. For example, science lessons 
typically do not occur every day. Since this is generally the case, the “Almost every day” response option 
can be interpreted literally (in which case it would never be selected) or can be interpreted as meaning 
“almost every day that we have science.” 

The latter interpretation is an example of an explicit time frame because the item behavior is 
explicitly linked to the period, “When you study science in school...” Since the respondent does not have 
science every day, the “almost every day” option cannot literally mean “almost every day.” It probably 
means “almost every day that I have science.” However, the other response options (“once or twice a 
week” and “once or twice a month”) implicitly link the behaviors to the calendar year. This creates a 
conflict between the item’s explicit and implicit time frame. If science lessons are provided three times a 
week and the focal behavior occurs about twice a week, respondents employing an implicit time frame 
will answer “almost every day” and respondents employing an explicit time frame will answer “once or 
twice a week.” 

Conflicts between explicit and implicit time frames can occur with other kinds of items. 

Consider a health survey item that asks: 

In the past twelve months , how often have you had the following health problems? Did you have 
the following problems daily or almost every day , once or twice a week , once or twice a month , or never 
or hardly ever? 

(a) Headaches 

(b) Upset Stomach 

(c) Morning sickness 

(d) Runny nose 

In responding to items a and b, most respondents would use the explicit time frame and report 
about their health over the past year. In responding to the item about morning sickness, very few women 
with morning sickness would use the explicit time frame (the past 12 months) for their rate estimates. 
They would respond with respect to the implicit time frame — the portion of their pregnancy when they 
had morning sickness. When a shift occurs in the implicit time frame, this shift may carry over to 
subsequent items. If explicit and implicit time frames are in conflict, interpretation of responses becomes 
challenging. 
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Item Comprehension 



The first stage of the item response process is comprehension and interpretation of the item’s 
meaning. If failure occurs at this stage — that is, if the respondent does not understand what the item is 
asking, there is a strong possibility of an inaccurate response. Several item comprehension problems 
were found with both fourth and eight grade students, as well as with teachers. 

Numerous words and phrases were not understood by many fourth graders. These included: 

♦ undecided 

♦ e.g. 

♦ novels 

♦ geometric shapes 

♦ telecommunications 

♦ science demonstration 

♦ practice problems 

♦ break words into parts 

♦ pulleys 

♦ chemicals 

♦ group activity or project 

Words and phrases were often interpreted in ways other than intended by the item writer. “Talk 
to the class about your mathematics work” generally evoked images of formal presentations in the minds 
of students. However, teachers would also think of less formal discussions. “Discuss solutions to 
mathematics problems with other students” was interpreted as cheating by at least four students. “Read 
on your own” could be interpreted as either referring to reading without anyone else’s assistance or as 
reading on your own volition. 

In some cases, attempts are made to facilitate comprehension of a technical term through the use 
of examples. It was noted that, in many cases, respondents would not generalize the construct. Instead, 
they would only respond to the examples provided. For example, one item asked whether students had 
ever done any hands-on activities or projects in school with any of the following: 

♦ Electricity (for example, batteries and flashlights) 

♦ Chemicals (for example, mixing or dissolving sugar or salt in water) 

Although some students could appropriately generalize these constructs, others would report 
about baking cakes or making lemonade as examples of activities with chemicals. 

Eighth grade students also had trouble understanding terms such as: 

♦ integrated or sequential mathematics 

♦ applied mathematics (technical preparation) 

♦ simple machines 

♦ levers 

♦ pulleys 

♦ portfolio 
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♦ content area 

♦ oral science report 

♦ design and carry out your own science investigation 

♦ practice problems 

♦ e.g. 

♦ science software 

♦ telecommunications 

♦ science demonstration 

Like their students, teachers did not have consistent interpretations of what was meant by 
“science demonstration.” 

Even students with good reading skills have difficulties with long, linguistically complex items. 
An example is an eighth grade mathematics item, where students were asked how much they agreed (on a 
five-point scale) with the following statement: “Describing mathematical concepts and ideas is as 
important as doing mathematical operations such as addition and multiplication in solving problems.” 
Half of the eighth graders (or 10 out of 20 students) checked the middle point “undecided” and one 
student skipped the item altogether. Seven of these students explicitly indicated that they chose 
“undecided” because they did not understand the item. 

A serious problem for teachers was their comprehension of the phrase, “students in your class.” 
This phrase was associated with two different kinds of comprehension problems. The first type of 
problem was associated with items such as: 

How often do the students in your class do each of the following? 

Do mathematics problems on worksheets? 

Talk to students about their mathematics work? 

For this item, “students in your class” was interpreted by some teachers as asking about their 
students’ behaviors, regardless of where they were performed. Other teachers interpreted the item as 
asking about behaviors performed by students while they were IN the classroom. 3 

More seriously (from a measurement perspective), it is unclear whether the teacher should be 
responding about the typical student or the frequency with which the behavior is performed by Miy 
student. The issue is moot when whole class activities are being reported upon. However, many of the 
items did not deal with whole class activities. This was clearly shown to be a problem for the following 
items: 



How often do the students in your class do each of the following? 

Use a computer? 

Some teachers respond for the typical student; others, for any student. The “any student” 
interpretation is more likely for rare events. For example, if a teacher is asked how often the students in 
her class punch or kick each other, it is very unlikely that (s)he will respond with respect to the typical 
student. 



3 We believe the latter interpretation (behaviors performed while in the classroom) is the intended interpretation. 
This belief is predicated on the assumption that the teacher and student questionnaires were attempting to measure 
identical constructs. Since the student item begins with the phrase, “When you do mathematics in school,” it is 
clear that the item is intended to measure within school behaviors. 
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Loss of Context 



Items that were presented in a list format (e.g., “How often do you do each of the following?”) 
produced problems because of lost context. That is, the respondents often forgot the stem and responded 
to the item as a stand-alone item. This problem was not restricted to fourth graders. For instance, five out 
of 20 eighth graders lost context when they were asked about the use of computers for mathematics at 
school. (See below.) 



When you do mathematics in school, how often do you do each of the following? 
Fill in one box on each line. 





Almost 


Once or 


Once or 


Never or 




every 


twice a 


twice a 


hardly 




day 


week 


month 


ever 


1. Do mathematics problems from textbooks 


□ 


□ 


□ 


□ 


2. Do mathematics problems on worksheets 
4. Solve mathematics problems with a 


□ 


□ 


□ 


□ 


partner or in small groups 
5. Work with objects like rulers, counting 


□ 


□ 


□ 


□ 


blocks, or geometric shapes 
6. Write a few sentences about how you 


□ 


□ 


□ 


□ 


solved a mathematics problem 


□ 


□ 


□ 


□ 


6. Take mathematics tests 

7. Talk to the class about your 


□ 


□ 


□ 


□ 


mathematics work 

8, Do 10 or more practice problems in 


□ 


□ 


□ 


□ 


mathematics by yourself 
9 . Discuss solutions to mathematics 


□ 


□ 


□ 


□ 


problems with other students 


□ 


□ 


□ 


□ 


10. Use a computer 


□ 


□ 


□ 


□ 



When these students responded, they had lost the mathematics context of the question. As a 
result, they overreported their computer usage by answering about their use of a computer anywhere and 
for any purpose. This finding was not due to probing effects (i.e., the interviewer asking probes after each 
question) since the students were instructed to answer all questions in a list before probing took place. 
Fourth grade students, eighth grade students, and teachers demonstrated loss of context to a variety of 
different items, indicating that this is a general survey design problem. 

The next section contains a listing of all of the questionnaire items investigated and a summary of 
responses to these items. When validating data were obtained from teachers, tables comparing students’ 
responses with their teacher’s responses are presented. The reasons for discrepancies are discussed, along 
with other issues that were noted. Recommendations for item modifications are also provided. 

Responses are summarized for fourth grade science items, followed by fourth grade mathematics 
items, fourth grade reading items, eighth grade science items, and eighth grade mathematics items. At the 
end of each of these sections we briefly discuss specific findings and present a tabular summary of 
discrepancy rates. 
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4. ITEM-BY-ITEM ANALYSES 



Fourth Grade Science Student and Teachers 

Student Item Only: 

1. How much do you agree with each of the following statements? Fill in one box on each line. 

Learning science is mostly memorizing. 

Agree □ 

Not sure □ 

Disagree □ 



Student Responses 


Frequency 


Agree 


15 


Not sure 


6 


Disagree 


2 



Discussion 

At least six students had very egocentric interpretations of this question. They interpreted the 
question as asking whether or not they agreed that they were able to remember the things they learned in 
science. For example, one student agreed that learning science is mostly memorizing because “I 
remember everything we did in science.” Another student disagreed because “I don’t remember a lot.” A 
third student responded “not sure” because “we may or may not remember what we did on a science 
project.” Other students misinterpreted the question. Below are some of the students’ interpretations of 
“mostly memorizing”: 

♦ That you can remember some of the answers from science 

♦ Science is mostly memorizing so you don’t have to forget something 

♦ If it’s easy to remember the science you study 

♦ If you can memorize what you do 

♦ They’re asking if I’m learning 

♦ Paying attention to what they tell you 

♦ It’s like something you’d need to know when you get to middle school 

Two students said “not sure” because they did not have science that often, and another student 
responded “not sure” because he did not understand the question. 
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Fourth Grade Science Student and Teachers 



Recommendations 

Many fourth grade students are not able to comprehend this question. They either interpret it as 
asking about themselves or misinterpret it altogether. Due to the high level of abstraction of the item, we 
suggest that this item not be administered to fourth graders. 
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Fourth Grade Science Student and Teachers 



Student Item: 

2. Have you ever done hands-on activities or projects in school with any of the following? Fill 
in all boxes that apply. 

Electricity (for example, batteries and flashlight bulbs)? 

Teacher Validation Probe (not a survey item): 

Have they done any activities in the area of electricity? 

Yes □ 

No □ 



Discrepancy rate 17% (4 out of 23 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 


19 


4 


Yes 







Discussion 

This question is a good example of one with seemingly good agreement between teachers and 
students, but with problems that surface during the student interview probes. 

Questions 2, 3 and 4 were very difficult for fourth graders to understand. The format (a single 
check box, to be checked if the option was true) was different from that of the other questions and so was 
confusing. There were many words used that are not in the average fourth grader’s vocabulary (i.e., 
electricity, apply, flashlights, bulbs, batteries). The format also leaves ambiguous the interpretation of the 
absence of a check for an item. Some students left the box unchecked because they did not do that 
activity. Other students left the box unchecked because they did not understand the question. Still other 
students checked the box when they did not understand the question. 

Fourth graders do not always attend to all of the subordinate clauses in an item. Some fourth 
graders focused on the word “ever” and reported activities outside their fourth grade science class. One 
student said he helped his Dad change light bulbs, and so checked this item. Another included activities 
in the third grade and so checked this item. A third student used a literal interpretation of “hands-on 
activities” and said he would never do hands-on activities because “that might be dangerous!” 

Specifying the context of “4 th grade science class” in the item may not be a sufficient solution. 
Teachers cover a wide range of content each year. When science is discussed, the teacher does not 
usually preface her presentation with “Now we will have a science lesson.” So, a lesson on nutrition may 
be a science lesson but might not be seen as science by the student. This issue seems to be associated with 
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Fourth Grade Science Student and Teachers 



frequent science underreporting by the student (relative to the teachers’ reporting of science behaviors and 
activities). 

A final general issue is that fourth graders often respond with respect only to the examples that 
are given. The examples become the focus of the question for fourth graders. Thus, examples need to be 
used sparingly. 



Recommendations 

The response options for questions 2, 3 and 4 should be changed to a “yes/no” format. The 
context of the item should be clarified to focus students’ answers on their 4 th grade activities: 

In your 4 h grade class , have you done activities or projects to learn about electricity? 

Yes □ 

No □ 

Even with such modifications, these questions may not be able to be reliably answered by fourth 
graders, unless the concepts can be greatly simplified. It might be wiser to ask the teacher if any of his or 
her students have done any electricity projects for science this year. 
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Fourth Grade Science Student and Teachers 



Student Item: 

3. Have you ever done hands-on activities or projects in school with any of the following? Fill 
in all boxes that apply. 

Chemicals (for example, mixing or dissolving sugar or salt in water)? 

Teacher Validation Probe (not a survey item): 

Have they done anything involving chemicals? 

Both: 



Yes □ 
No □ 



Discrepancy rate 



70% (16 out of 23 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 


6 


16 


Yes 




1 



Discussion 

Students seemed to understand the concept of the question. Several appropriate examples of 
activities related to chemistry were given. However, some may reflect an exclusive focus on the examples 
used in the item: 

♦ “baking a cake using sugar and salt” 

♦ “we pour in what they tell us to pour - we did sugar crystals with sewage, hot water and a 
stick” 

♦ “dissolving sugar in water” 

♦ “making crystals” 

♦ a popcorn experiment 

♦ activities with sugar, salt and water 

♦ “My mom made me lemonade this morning” 

♦ “using a stethoscope to listen underwater” 

As with the previous question, some students took a literal interpretation of the word “ever” and 
checked “Yes” to the item when they had done chemical activities in other grades and outside of class. 
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Fourth Grade Science Student and Teachers 



A final source of discrepancy between the students’ and teachers’ answers may be due to an 
incorrect response on the part of the teacher. At least two teachers used science kits where vinegar was 
used on rocks to test for calcite. Some students from both classes listed this activity. However, only one 
teacher remembered the activity or categorized it as an activity involving chemicals. 



Recommendations 

Ask the teacher if his/her students have done any chemistry activities or projects this year. 



O 
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Fourth Grade Science Student and Teachers 



Student Item: 

4. Have you ever done hands-on activities or projects in school with any of the following? Fill 
in all boxes that apply. 

Simple machines (for example, pulleys and levers)? 

Teacher Validation Probe (not a survey item): 

Have they done anything with simple machines? 

Both: 



Yes □ 
No □ 



Discrepancy rate 



13% (3 out of 23 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 


20 


3 


Yes 







Discussion 

None of the teachers reported doing activities with simple machines. However, three students 
said they had done activities with simple machines. One student said he weighed rocks, used tape 
measures and nails and interpreted these to be activities with simple machines (weights and a tape 
measure). The other two students reported on activities using a literal interpretation of “have you ever”: 
activities done in the second grade and a robot made “one time” that walked by itself with a LEGO 
machine set. The unanchored time context and the difficult words (very few of the students understood 
what pulleys and levers were) contributed to the overreporting by students. 



Recommendations 

This activity happens infrequently and will be difficult to make comprehensible to fourth graders. 
It may be easiest to simply drop this item or ask the teacher about the activity. 



BEST COPY AVAILABLE 
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Fourth Grade Science Student and Teachers 



Student Item: 

5. About how much time do you usually spend each day on science homework? 



I don’t have science □ 

None □ 

V 2 hour □ 

1 hour □ 

2 hours □ 

More than 2 hours □ 



Teacher Validation Probe (not a survey item): 

About how long do you think it takes your students to do their science homework? 

minutes per assignment 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 
Students reporting they didn’t have science 



67% (14 out of 21 responses; 2 missing) 
5 
0 
9 



Teacher 

Responses 


Student Responses 


1 don’t have 
science 


None 


V 2 hour 


1 hour 


2 hours 


More than 
2 hours 


0 


9 


7 


3 


1 






15 minutes 










1 




30 minutes 














1 hour 














2 hours 














More than 2 
hours 
















4 0 
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Fourth Grade Science Student and Teachers 



Discussion 

There were a number of problems with this item: 

♦ Most of the students misinterpreted the first response option “I don’t have science” as “I 
don’t have science homework.” All agreed that they did have a science class. Several of the 
students choosing “none” saw that option as identical to “I don’t have science” which, again, 
they read as “I don’t have science homework.” 

♦ Some students also took an evaluative interpretation of the options. One student interpreted 
the “none” option as “you don’t do your homework.” Other students felt the options 
measured their abilities - that the amount of time spent on homework reflected how difficult 
or easy the homework was for them. 

♦ Many of the students who chose the non- zero options were responding in terms of homework 
in general, combined across subject areas. Similarly, they were not sure which homework 
counted as science homework. Two students asked if their social studies homework should 
be included. 

♦ Many students did not understand the fraction “Vi hour” and misread it as “1 to 2 hours.” 
Thus they would pick “none” or “1 hour” as the smallest options they saw. 

♦ Finally, students who correctly interpreted the question (e.g., “I look at the clock to see how 
much time I spend doing science homework”), and understood each response option were 
blocked when their science homework amounted to less than Vz hour each week. So, when 
they expressed spending about 5-10 minutes a week on science homework, they chose “Vi 
hour” as the smallest non-zero option. 

Recommendations 

Rephrasing and offering different choices in response options may help clarify them for fourth 
graders. Rephrasing the question may help clarify for fourth graders that the question is about science 



lessons: 



About how much time do you spend doing science homework ? 



I am not taking science this year 

The teacher doesn’t give us science homework 

About 15 minutes a day doing science homework 

About 30 minutes a day doing science homework 

About 1 hour a day 

About 2 hours a day 

More than 2 hours a day 



□ 

□ 

□ 

□ 

□ (Since few students , if any, spend 

□ more than 15 to 30 minutes doing 

□ science homework, these 3 options may 
be deleted - or replaced with “more than 
45 minutes a day. ”) 
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Fourth Grade Science Student and Teachers 



Student Item: 

6. In the last two years, have you participated in a science fair, festival, or special science day? 

Yes □ 

No □ 

Teacher Validation Probe (not a survey item): 

About how many of your students participated in a science fair, festival, or special science 
day in the last two years? 

students 



Discrepancy rate 



43% (10 out of 23 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 

(0 students) 


11 


5 


Yes (any # of 
students) 


5 


2 



Discussion 

Two teachers responded that at least one of their students participated in a science fair, festival, or 
special science day in the last two years. The students of one teacher participated in a science open house 
last year. The other teacher said that they had a science fair for fourth graders last year, but this year the 
school building was under construction and there was not enough space to house the displays. The 
students who said “no” when this teacher said “yes” were confused as to the context of the question. One 
student interpreted the question narrowly to mean: “when you share a science project.” One student was 
referring to only the past year and ignored the introductory phrase “in the last two years.” One student 
seemed to understand the question, but did not participate with the other students in the science fair. 

The five students who said they had participated in a science fair in the last two years, when their 
teachers said there had been no such activity misunderstood the intent of the question. For example, one 
student said “I went to a festival where I played baseball and kickball 2 years ago,” another said “I went 
to a science fair in January in the cafeteria. It was for kids in 5 th , 6 th , 7 th , and 8 th grade. I went in to see it 
for one minute and went in a circle to see activities.” He did not present at the fair; he considered his 
attendance at the fair to be participation. Although this might be a valid interpretation, it does not appear 
to be the interpretation intended by the writer of the item. 
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Fourth Grade Science Student and Teachers 



Recommendations 

Changing the wording of the question may help to clarify the context. However, students have 
such varying views of “science fair” that it may be difficult to get students all to answer in terms of the 
same concept. If the concept includes participation and excludes visiting without participating, then the 
following question may be easier for fourth graders to understand: 

In the last two years , have you ever done a science project that was shown at a science fair, 
science festival, or special science day? 

Yes □ 

No □ 
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Fourth Grade Science Student and Teachers 



Student Item: 

7. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Teacher Item: 

These questions refer to your science instruction in general. About how often do your science 
students do each of the following? Fill in one box on each line. 



Both: 



Discuss science in the news? 



Original Scale (both): 

Almost every day 
Once or twice a week 
Once or twice a month 
Never or hardly ever 





Revised Scale: 




□ 


Almost every day 


□ 


□ 


Sometimes 


□ 


□ 


Never 


□ 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



69% (11 out of 16 responses) 
6 
5 



Teacher 

Responses 


Student Res 


ponses 


Never or hardly 
ever 


1-2 times/ 
month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


4 


2 


2 




1-2 times/ month 






1 




1-2 times/ week 


5 




1 


1 


Almost every day 
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Fourth Grade Science Student and Teachers 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



43% (3 out of 7 responses) 
1 
2 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every day 


Never or hardly ever 


1 


1 




1-2 times/ month 


1 


1 




1-2 times/ week 


1 


2 




Almost every day 









Discussion 

Students’ interpretations of the phrase “science in the news” varied widely. Many thought of the 
media. These students saw “science in the news” as referring to others in the media discussing science: 

♦ “the news - what you watch on TV; they do experiments that they have tried” 

♦ “if you heard it in the news” 

♦ “if I watched the news on television” 

♦ “news about science in the newspaper; when they do in science when they read the news” 

♦ “the news people” 

♦ “if there is science in the news (I don’t watch the news)” 

♦ “watching TV and science programs” 

♦ “I watch Mysteries of Science on TV” 

or as whether or not the student has been in the media discussing science: 

♦ “like playing that you were a news reporter” 

♦ “if you’ve ever been on the news talking about science” 

However, some decided that it was sharing or listening: 

♦ hearing about science but not talking about it 

♦ “sharing what you did with the class” (as news) 

For both the 3- and 4-point response scales, the overreporting and underreporting was about 
equal. There was not one consistent conceptualization of “science in the news” by students. Even 
teachers did not consistently define the construct. Thus, the responses were almost random. 




4 
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Fourth Grade Science Student and Teachers 



Recommendations 

The definition of “science in the news” needs to be specified. For example, if the desire is to 
know how often teachers try to find real-life examples for science topics, then the item might be 
reworded. Since the students often cannot distinguish science instruction from other instruction, this item 
should only be administered to teachers: 

How often do you talk to your class about something in science that you heard on radio or 
television or read in newspapers or magazines? 

Almost every day □ 

Sometimes □ 

Never □ 

(In order to keep the item simple, it is assumed that, when news events are discussed, the teacher tries to 
relate them back to the topic at hand. Also, the 3-point scale does seem to be easier for students to use, 
especially for items such as this, which are usually done on an irregular basis.) 
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Fourth-Grade Science Student and Teachers 



Student Item: 

8. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Do hands-on activities in science? 

Teacher Item: 

These questions refer to your science instruction in general. About how often do your science 
students do each of the following? Fill in one box on each line. 



Do hands-on activities or investigations in science? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



75% (12 out of 18 responses) 
4 
8 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 


3 


2 


1 


1 


1-2 times/ week 


2 


3 


2 


2 


Almost every day 











BEST COPY AVAILABL 
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47 



Fourth-Grade Science Student and Teachers 



Discrepancy rate 



0% (0 out of 7 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 0 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 




3 




1-2 times/ week 




4 




Almost every day 









Discussion 

This item makes salient important issues associated with asking science behavioral frequency 
items of fourth graders. For this item, teachers either responded “1-2 times per week” or “1-2 times per 
month.” They explained that the activity was, by and large, sporadic, depending on whether or not they 
were using science kits, or working on a specific project. Most of the teachers averaged over the year, but 
doubted that their students could do this. And, in fact, most of the students, when they did understand the 
question, responded primarily in terms of recent projects and frequencies. 

Lack of understanding and subsequent guessing contributed to most of the underreporting of 
students. When they did not know what “hands-on activities” meant, they tended to choose “never or 
hardly ever.” One student felt her “class does not do science” and there was no option for this answer. 
One student felt that hands on activities were only done at home. 

Even the zero error-rate for the group using the 3-point scale did not reflect a true correspondence 
between students’ and teachers’ answers. The students who responded “sometimes” didn’t understand 
the question, gave examples of hands-on activities at home, or thought only of one recent activity. 

Finally, one student who did seem to have the same definition of “hands-on activities” as his teacher 
answered: “every day we use the kits, we do hands-on activities almost every day.” 



Recommendations 

Any revised version of the items needs to overcome two problems: conveying to teachers and 
students a similar concept of “hands-on activities” and getting students to average over time, or at least 
getting teachers and students to use a similar time reference period. This item should probably just be 
asked of teachers. 
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Fourth-Grade Science Student and Teachers 



Student Item: 

9. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Talk about measurements and results from your hands-on activities? 

Teacher Item: 

These questions refer to your science instruction in general. About how often do your science 
students do each of the following? Fill in one box on each line. 

Talk about measurements and results from students’ hands-on activities? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



75% (12 out of 16 responses) 

3 

9 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 


4 


2 


1 


1 


1-2 times/ week 


2 


' 3 


2 


1 


Almost every day 
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Fourth-Grade Science Student and Teachers 



Discrepancy rate 



29% (2 out of 7 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 2 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 


1 


3 




1-2 times/ week 


1 


2 




Almost every day 









Discussion 

This item has many sub-parts and is quite complex. Students needed to combine the concepts of 
measurements in science, results in science, talking about measurements and results and hands-on 
activities, to retain the context of their present 4 th grade science class, and to ascertain that the activity 
could include formal presentations in front of class, informal discussions, or discussions with their 
teacher. This item was simply beyond the comprehension level of most of the students. Even teachers 
had different interpretations of the item: 

♦ only teacher-led discussions or formal presentations should be included 

♦ questions to the class and informal conversations should be included 

Nine students, when probed, responded only that they had no idea what the question was asking. 
Of those, four responded “never or hardly ever” or “never,” three responded “1-2 times/month,” one 
responded to the 3-point scale “sometimes” and one responded “almost every day.” This shows that 
students use many of the response options, even when they are guessing. 

Students gave some very interesting examples of what was meant by “talk about measurements 
and results from your hands-on activities:” 

♦ “grab a ruler and measuring it with your bare hands, like measuring an inch” 

♦ “talking about measuring paper to draw inches in math” 

♦ “they want to know did you do measurements at school or at home” 

♦ “we never talk about measurements about our hands in science” 

♦ “using rulers to measure, plotting X and Y axis” 

♦ “how much I stand up in front of the class and tell them about my experiments and 

♦ inventions. I usually don’t talk about this stuff ‘cause I like to keep my inventions secret!” 

♦ “we talk about measurement of something” 

♦ “getting a ruler or something, you measure something, it should come out the way you 

♦ want it to come out” 
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Fourth-Grade Science Student and Teachers 



Recommendations 

Delete the item or ask as a shorter question with a clearly specified context: 

After a science demonstration , do you talk about what happened with other students? 

Yes □ 

No □ 



O 
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Fourth-Grade Science Student and Teachers 



Student Item: 

10. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Use a computer for science? 

Teacher Item: 

These questions refer to your science instruction in general. About how often do your science 
students do each of the following? Fill in one box on each line. 



Use computers for science? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



13% (2 out of 16 responses) 

1 

1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 


14 


1 






1-2 times/ month 


1 








1-2 times/ week 










Almost every day 
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Fourth-Grade Science Student and Teachers 



Discrepancy rate 



57% (4 out of 7 responses) 



Number children reporting more frequently 1 

Number children reporting less frequently 3 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 


2 




1 


1-2 times/ month 


3 


1 




1-2 times/ week 








Almost every day 









Discussion 

One teacher’s initial response to this item was “never or hardly ever.” In the past week, it was 
used one (out of three) science days. Accordingly, the teacher adjusted her estimate to “once or twice a 
month.” The teacher explained that due to the completion of construction, her class had recently moved 
from a portable into a regular classroom. When in the portable, it was more difficult to use resources 
such as computers. Thus, in her initial response she was thinking of instruction in the portable, as 
opposed to the classroom. Four of her students responded “never or hardly ever.” They might have been 
thinking of their science instruction in the portable, as well. 

One student lost context and overreported his computer usage in science. He uses a computer at 
home “almost every day,” but not necessarily for science. Thus, he not only lost the context of “when 
you study science in school ” but also the use of a computer for science only. Another student had used a 
computer for a science project on Hawaii a few weeks earlier. Because this event was salient in his mind, 
the student chose “once or twice a month” as opposed to “never or hardly ever,” which was his teacher’s 
response. This is an example of Tversky’s availability heuristic - if information about a type of activity is 
easily accessible in memory, the individual will assume this ease of accessibility is indicative of the fact 
that the event occurs frequently. 

One teacher’s initial response was “never or hardly ever.” In the past week, it was used four (out 
of four) days. The teacher adjusted her estimate to “once or twice a month.” Her initial estimate did not 
consider the use of computers as library resources, to get information about science research projects. 

Another teacher did not perceive any difference between this item (“About how often do your 
science students use computers for science?”) and item 27 (“When you teach science, about how often do 
you use computers for science (e.g., science software, telecommunications)?”). Thus, the teacher lost the 
context of the questions, not realizing that one question asked about her students’ behavior and the other 
about her own behavior. 

Recommendations 

The item appears to work relatively well for students. However, the use of computers in science 
does not seem to be a very common instructional tool at the fourth grade level. Response variance is 
minimal. 



The Measurement of Instructional Background Indicators 



45 



Fourth-Grade Science Student and Teachers 



The teacher item could be improved. It should be clarified whether the use of computers as 
library resources is supposed to be included in the teachers’ responses or not. In addition, the first part of 
the stem: “These questions refer to your science instruction in general” should be deleted to focus 
teachers on their students’ behaviors. The stem would then read as follows: 

About how often do your science students do each of the following? Fill in one box on each 
line . 
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Fourth-Grade Science Student and Teachers 



Student Item: 

11. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Teacher Item: 

These questions refer to your science instruction in general. About how often do your science 
students do each of the following? Fill in one box on each line. 



Both: 



Use library resources for science? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



69% (1 1 out of 16 responses) 

6 

5 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 


1 


2 






1-2 times/ month 


5 


4 


3 


1 


1-2 times/ week 










Almost every day 
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Fourth-Grade Science Student and Teachers 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



29% (2 out of 7 responses) 
2 
0 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 


4 


2 




1-2 times/ month 




1 




1-2 times/ week 








Almost every day 









Discussion 

A general issue relating to most of the science items is: What is science to a fourth grader? 
Teachers cover a wide range of content each year. When science is discussed, the teacher probably does 
not preface her presentation with “Now we will have a science lesson.” So, a lesson on nutrition may be 
a science lesson but might not be seen as science by the student. Similarly, a lesson on geology may be 
perceived as “learning about rocks” and not linked to any broader subject area by a fourth grader. (There 
is little reason for them to make such linkages.) Furthermore, reading science news in a magazine might 
be considered as being reading rather than science. This issue seems to be associated with frequent 
science underreporting by the student (relative to the teacher’s reporting of science behaviors and 
activities). 

Some of the underreporting was due to the issue discussed above. Another source of both under- 
and overreporting was the term “library resources.” At least five students either did not know what the 
term meant or did not have a clear understanding of it. In addition, at least two students lost the science 
context of the question. One student said: “We don’t have a library, but we have books and we use them 
every day.” Another student explained: “If I don’t have books I go to the library so I can do my 
homework.” However, this student does not have science homework. 

Finally, one student included the use of a library for science at another school last year. This 
interpretation is legitimate given the unanchored timeframe of the question. 

One teacher included two sets of encyclopedias in the classroom for the students to use, as well as 
CD-ROMs as library resources. Another teacher did not include the public library in her response. 



Recommendations 

The term “library resources” is not clear to fourth graders. The term should either be clarified or 
the item not administered to fourth grade students. The term is not clear to teachers either: Should library 
resources (e.g., encyclopedias and CD-ROMs) in the classroom be included? And what about science 
resources from public libraries - should they be included in the teachers’ responses? Clear instructions 
need to be provided. 
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Fourth-Grade Science Student and Teachers 



Student Item: 

12. When you study science in school, how often does your teacher do each of the following? 
Fill in one box on each line. 

Teacher Item: 

When you teach science, about how often do you do each of the following? Fill in one box on 
each line. 



Both: 



Talk to the class about science? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



81% (13 out of 16 responses) 

7 

6 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 










1-2 times/ week 


3 


3 


3 


7 


Almost every day 
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Fourth-Grade Science Student and Teachers 



Discrepancy rate 



29% (2 out of 7 responses) 



Number children reporting more frequently 1 

Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 








1-2 times/ week 


1 


4 


1 


Almost every day 






1 



Discussion 

At least five students had problems defining science. Students either underreported because they 
did not recognize when they were being taught science (see discussion on previous item), or they 
overreported because they defined science very broadly (e.g., as geography, history, and family). 

In one class, science teaching was only associated with use of “Foss kits,” also referred to as 
units. At other times, science time was used to teach students social skills. When kits were used, science 
was taught daily. The teacher’s estimates for this item was modified based on review of the past week, 
because when the teacher responded she was thinking about the weeks in which the science units were 
employed. However, averaged over the year, these activities were engaged in less frequently. So, she 
changed her response from “almost every day” to “once or twice a week.” However, at least three of her 
students were not able to average science teaching over the course of the school year and therefore 
responded “almost every day,” which led to overreporting. These students appear to have interpreted this 
item to be asking about the days that science was taught. That is, “When your teacher teaches science, 
how often does she talk to the class about science?” Such an interpretation is reasonable, particularly 
when asking about rare events. 

In another class, science teaching also revolved around units. These units were used about every 
six weeks. Science was not taught every day. For example, last week science was only taught on two 
days — and this was the food pyramid. The teacher did not immediately think of the food pyramid and 
nutrition as science. This strongly suggests that science teaching, particularly in states without a science 
curriculum will be difficult to define. Furthermore, regardless of whether or not there is a curriculum, 
students will have problems recognizing the fact that they are being taught something “as part of science.” 

Two students had problems defining “talk to the class about science,” which led to 
underreporting. They defined the term in the following ways: 

♦ The teacher just tells us what to do, she doesn’t tell us what it is. 

♦ The teacher doesn’t tell us about science, we just do the projects. 

Finally, one student ignored the stem of the question and responded with respect to how often she 
(the student) talked to the class about science. 
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Fourth-Grade Science Student and Teachers 



Recommendations 

Fourth graders have trouble defining “science,” the basic construct of the question. Errors arise 
when they do not recognize when they are being taught science or define science too broadly. They also 
have difficulty averaging instructional behaviors in science over time, because science often is taught in 
units, as opposed to on a daily basis. This also creates problems for teachers, who are a better source of 
information about this than are fourth graders. 
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Fourth-Grade Science Student and Teachers 



Student Item: 

13. When you study science in school, how often does your teacher do each of the following? 
Fill in one box on each line. 

Teacher Item: 

T4S25. When you teach science, about how often do you do each of the following? Fill in one 
box on each line. 



Both: 



Do a science demonstration? 

Original scale (both): 

Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Revised scale: 

Almost every day □ 

Sometimes □ 

Never □ 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



88% (14 out of 16 responses) 
4 
10 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


1 




1 




1-2 times/ month 


1 








1-2 times/ week 


8 


1 


1 


3 


Almost every day 











BEST COPY AVAILABLE 
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Fourth-Grade Science Student and Teachers 



Discrepancy rate 



29% (2 out of 7 responses) 



Number children reporting more frequently 1 

Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 


1 


i 




1-2 times/ month 


1 


4 




1-2 times/ week 








Almost every day 









Discussion 

There were several sources of discrepancies between the students’ and teachers’ responses. At 
least five students lost the context of the question. That is, they interpreted the question as asking about 
how often they (the students) did science demonstrations. Below are some of their interpretations: 

♦ We never demonstrate science. The teacher just tells us what to do. 

♦ I don’t because I never do. I raise my hand but the teacher never calls on me, and I might 
make a mistake and everyone would laugh at me. 

♦ When we do something, we have to show it to the class. 

Another source of discrepancy was the lack of understanding of the term “science 
demonstration.” At least four students did not have a clear understanding of this term. 

Four students underreported their teachers’ behavior because they did not recognize when they 
were being taught science (see previous discussion). 

Another three students overreported their teachers’ behavior because they interpreted the question 
as: “When we have science, how often does the teacher do a science demonstration?” Thus, they were not 
able to average science teaching over the course of the school year. 

Some of the teachers were not clear on how to define “science demonstration.” One teacher 
included student participation in the science demonstrations, whereas another teacher did not. A third 
teacher included setting things up for students to enable them to do the activity as part of the science 
demonstration. 



Recommendations 

This question is too hard for fourth grade students to answer. They either lose context, do not 
understand the constructs of “science demonstration” and “science,” or are unable to average science 
teaching over the course of the school year due to the relatively infrequent occurrences of science 
teaching. This item should not be administered to fourth graders. 

The intent of the teacher item has to be clarified, with a clear definition of “science 
demonstration.” 
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Fourth-Grade Science Student and Teachers 



Student Item: 

14. When you study science in school, how often does your teacher do each of the following? 
Fill in one box on each line. 

Teacher Item: 

When you teach science, about how often do you do each of the following? Fill in one box on 
each line. 



Both: 



Show a science videotape or science television program? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



63% (10 out of 16 responses) 
10 
0 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


2 


5 


1 


1 


1-2 times/ month 




4 


3 




1-2 times/ week 










Almost every day 
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Fourth-Grade Science Student and Teachers 



Discrepancy rate 



71% (5 out of 7 responses) 



Number children reporting more frequently 2 

Number children reporting less frequently 3 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 




2 




1-2 times/ month 


3 


2 




1-2 times/ week 








Almost every day 









Discussion 

At least eight students had trouble estimating the frequency with which their teacher showed a 
science videotape or science television program. Three students interpreted the question as asking: 
“When we have science, how often does the teacher show a science videotape or science television 
program?” That is, rather than averaging over a week or a month, the implicit denominator for their rate 
calculations were the days on which science was offered. This lead to overreporting. 

Three students misinterpreted the question. One student indicated that the question made no 
sense to him. He thought it was asking how often he or other students show videotapes. He said it was 
asking: “If you did a science thing on videotape.” Another student included watching science programs 
(Mysteries of Science) at home in his response. Finally, a third student responded “never or hardly ever” 
because “the class is too noisy, three boys always interrupt, we don’t get finished with our lesson. We 
never get videotaped because people make faces.” Thus, she interpreted the question as the class being 
videotaped as opposed to the class watching videotapes. 



Recommendations 

Fourth graders have a hard time estimating the frequency with which their teacher shows a 
science videotape or science television program. They tend to overreport this behavior by their teacher. 
Some students also misinterpret the question. They either forget that the question is asking about their 
teacher, as opposed to themselves, or include science programs they watch at home. If this item is asked, 
it should be explicitly linked to days on which there is science instruction. 



The Measurement of Instructional Background Indicators 



55 



Fourth-Grade Science Student and Teachers 



Student Item: 

15. When you study science in school, how often does your teacher do each of the following? 
Fill in one box on each line. 

Use computers for science (e.g./such as, science software, telecommunications)? 

Teacher Item: 

When you teach science, about how often do you do each of the following? Fill in one box on 
each line. 

Use computers for science (e.g., science software, telecommunications)? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 13% (2 out of 15 responses) (1 missing) 

Number children reporting more frequently 1 

Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 


13 


1 






1-2 times/ month 


1 








1-2 times/ week 










Almost every day 












The Measurement of Instructional Background Indicators 



56 



Fourth-Grade Science Student and Teachers 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



33% (2 out of 6 responses) (1 missing) 
0 
2 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 


2 






1-2 times/ month 


2 


t ■ 




1-2 times/ week 








Almost every day 









Discussion 

In spite of the low discrepancy rates, this item had problems. The first 1 1 students answered a 
version of the question that included “e.g.”: “Use computers for science (e.g., science software, 
telecommunications)?” However, at least five of these students did not know what “e.g.” meant. So it 
was decided to change “e.g.” to “such as,” which worked much better. 

An even bigger problem was the word “telecommunications.” At least nine students did not 
know what it meant. As one student put it: “Like a fourth grader can read that! My Mom and Dad 
probably can't read it!” Another student checked “never or hardly ever” because “she [the teacher] never 
taught us that word.” When students attempted to define it, they came up with the following: 

♦ On TV they show a web site 

♦ Talk with the person in the computer 

♦ Doing research on the computer 

♦ When you talk to other people about science [through a computer as opposed to in groups] 

The term “science software” also created problems. At least three students did not know what it 
meant. Some students defined it as: 

♦ A disk of science 

♦ Use a computer for software 

♦ Looking up stuff on disc and saving it 

♦ Like the movie [video] we saw 
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Fourth-Grade Science Student and Teachers 



Recommendations 

This item is an example of how examples in a parenthesis can hurt much more than help. The 
item would work much better for fourth graders without the examples: 

Does your teacher ever use a computer to teach you science? 

Yes □ 

No □ 



O 
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Fourth-Grade Science Student and Teachers 



Student Item: 

16. About how often does your science class go on a science field trip? 

3 or more times a year □ 

1 or 2 times a year □ 

Never or hardly ever □ 

Teacher Item: 

About how often do your science students go on a science field trip? 

3 or more times a year □ 

1 or 2 times a year □ 

Never or hardly ever □ 



Discrepancy rate 



39% (9 out of 23 responses) 



Number children reporting more frequently 4 

Number children reporting less frequently 5 



Teacher 

Responses 


Student Responses 


Never or 
hardly ever 


1 or 2 times 
a year 


3 or more 
times a year 


Never or hardly ever 


5 


2 


i 


1 or 2 times a year 


5 


9 


i 


3 or more times a year 









Discussion 

The term “science field trip” is not clear to some students. Four students overreported going on 
science field trips because they included field trips to missions, Indian reservations, and going to the 
library to read science books. In addition, one of these students considered a field trip where they had 
discussed science-related issues as a science field trip. Another student’s class had gone to NASA, but 
the student did not consider this to be a science field trip and therefore responded “never or hardly ever.” 
When asked for an example of a science field trip, he replied “Going to a place like this [Cognitive 
Survey Lab] or going to labs at Stanford Hospital [where his mom works].” Two other students in his 
class who had been to NASA responded similarly (“never or hardly ever”). 

Even teachers had trouble defining “science field trips.” One teacher correctly decided that a 
field trip whose primary purpose was science is a science field trip. She thought of other field trips with 
science components (e.g., stopping off to see a seismograph and learning about candle-making), but she 
would not consider these to be science field trips. However, another teacher responded “1 or 2 times a 
year,” but provided examples that were not science field trips. They went to Mission San Juan Bautista. 
She was responding to “field trips” and wanted to report all field trips, assuming this was our interest. 
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Fourth-Grade Science Student and Teachers 



Only one of the teacher’s science students was interviewed and the student provided the same response as 
the teacher because she counted a science trip they went on last year. 

Finally, one student responded that the class went on a field trip for science this year. (According 
to the teacher, they had not gone on one yet, but would be going to the planetarium. However, this fact is 
not relevant to the issue that was uncovered.) This student categorized this one trip as “never or hardly 
ever” rather than as “1 or 2 times a year.” Although the item is asking about discrete events, the scale 
includes both vague quantifiers (“never or hardly ever”) and precise quantifiers (“1 or 2 times a year,” “3 
or more times a year”). The vague quantifier “hardly ever” clearly refers to things which have occurred. 
So, “never or hardly ever” was logically interpreted as 0 or 1 time and a “one trip” response was classified 
as being part of this response option. We would recommend that the final response option be changed to 
a precise quantifier (i.e., “Never”). 



Recommendations 

Some students cannot clearly define a science field trip. This relates to difficulties fourth graders 
have defining “science,” as discussed earlier. If the item is administered to students, the scale should be 
changed, as discussed above. Additionally, the item should be anchored to the current school year: 

This school year, how many science field trips did your class go on? 

None □ 

1 or 2 □ 

3 or more □ 

Similar changes should be made to the teacher item, as well. 
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Fourth-Grade Science Student and Teachers 



Teacher Item Only: 

T28. Counting this year, how many years in total have you taught the following subjects? 
(Include any full-time or part-time assignments, but not substitute assignments.) 

Science 



2 years or less □ 

3-5 years □ 

6-10 years □ 

11-24 years □ 



25 years or more □ 



Teacher Responses 


Frequency 


2 years or less 


i 


3-5 years 


2 


6-10 years 




1 1-24 years 


1 


25 years or more 


2 



Discussion 

One teacher had a more inclusive interpretation of science as opposed to math. (Math was the 
first item in the list. It is written up in the fourth grade math section.) She included home economics as 
part of her science experience but not as part of her math experience, even though students use fractions 
such as half cups in their cooking lessons. Accordingly, she reported more years of science than math 
teaching experience. 

Conversely, another teacher reported less science than math teaching experience. Even though 
her career was almost exclusively at the elementary grades, she said that she taught science for 3 - 5 years 
and had taught math 1 1-24 years. She did not consider herself to be very well prepared to teach most 
areas in science. Science teaching for most of her career was minimal - the science lessons that she 
prepared and taught were rare events. Science was not in the curriculum. She completed preparation for 
teaching a science curriculum/program a few years ago; her teaching subsequent to using the new 
program was not considered as science teaching. Her previous science pedagogical experiences were too 
episodic and of too poor a quality for her to count them as time spent teaching science. 

These interpretations appear to be idiosyncratic and overall the item worked as intended. 



Recommendations 

None. Keep the item as is. 
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Teacher Item Only: 

T29. How often do you use each of the following to assess student progress in science? Fill in 
one box on each line. 

Multiple-choice tests 



Once or twice a week □ 

Once or twice a month □ 

Once per grading period □ 

Once or twice a year □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Once or twice a week 




Once or twice a month 


1 


Once per grading period 


1 


Once or twice a year 




Never or hardly ever 


4 



Discussion 

The item is asking about discrete events — that is, things that can only take on integer values. 
However, the scale employs both specific and vague quantifiers. “Never or hardly ever” means 
something that happens zero or one time per year. “Once or twice a year” means something that occurs 
one or two times per year. So, something that occurs once a year can be classified in either category. 

One teacher indicated that she had three grading periods in the year. Her class was presently in 
the second grading period. Accordingly, for her something that had happened once so far could be 
answered as either “Never or hardly ever,” “Once or twice a year,” or “Once per grading period”! 

The length of a grading period can be idiosyncratic. Without knowing the exact length of a 
grading period, responses may be difficult or impossible to interpret. One teacher stated that she 
generally gives one test per unit. Units typically last six weeks. Their grading period is nine weeks. 
Since this is more than “c - once per grading period,” she chose “b - once or twice a month” as her 
response to item 29 (since all of her tests include multiple choice items). 

For items 29 and 30, another teacher initially responded “Once per grading period.” This 
appeared to be a socially appropriate response. She indicated later that she really had not given the 
students any tests. Since many of her students take science with other teachers (in Spanish), “testing 
would have to be done cooperatively. So, we really haven’t had any tests.” Similarly, for an item asking 
about homework, the teacher indicated that she really does not give science homework. She thinks she 
might have given one homework assignment. By the time she responded to “Homework,” it was clear 
that the linkage to “to assess student progress in science” had been lost. 
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Recommendations 



Change the scale: 

Once or twice a week □ 

Once or twice a month □ 

Three or four times a year □ 

Once or twice a year □ 

Never □ 
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Teacher Item Only: 

T30. How often do you use each of the following to assess student progress in science? Fill in 
one box on each line. 

Short or long written responses (e.g., a phrase or sentence; or several sentences or 
paragraphs) 



Once or twice a week □ 

Once or twice a month □ 

Once per grading period □ 

Once or twice a year □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Once or twice a week 


2 


Once or twice a month 


1 


Once per grading period 


2 


Once or twice a year 




Never or hardly ever 


1 



Discussion 

One teacher had only given one science test this year, but students write short or long written 
responses every day. However, these are rarely looked at. So, she estimated that per student, she uses 
this information about “once or twice a month.” 

Another teacher does not use short or long written responses for tests, since she does not give 
science tests. However, she has her students write in their “Science Journal” every day, so she chose 
“once or twice a week.” 

Also, see discussion about issues with the rating scale in item 29. 



Recommendations 

It is not clear if this item refers to tests, assignments, or both. The intent (of which we are 
uncertain) needs clarification. Nonetheless, the rating scale should be modified, as suggested in item 29. 
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Teacher Item Only: 

T31. How often do you use each of the following to assess student progress in science? Fill in 
one box on each line. 

Portfolio collections of each student’s work? 



Once or twice a week □ 

Once or twice a month □ 

Once per grading period □ 

Once or twice a year □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Once or twice a week 




Once or twice a month 


1 


Once per grading period 


2 


Once or twice a year 


3 


Never or hardly ever 





Discussion 

It appeared that the teachers understood the term “portfolio collections”. 
There is a problem with the rating scale, as discussed in item 29. 

Recommendations 

The rating scale should be modified, as suggested in item 29. 
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Teacher Item Only: 

T32. How often do you use each of the following to assess student progress in science? Fill in 
one box on each line. 

Homework 



Once or twice a week □ 

Once or twice a month □ 

Once per grading period □ 

Once or twice a year □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Once or twice a week 




Once or twice a month 




Once per grading period 


1 


Once or twice a year 


1 


Never or hardly ever 


4 



Discussion 

Two of the six teachers lost the context of the question. That is, they lost the linkage to “to assess 
student progress in science.” They both interpreted the item as: “How often do I give science 
homework?” However, since one of the teachers never gives homework, her answer was “never or hardly 
ever.” 



There is a problem with the rating scale, as discussed in item 29. 



Recommendations 

In order to avoid the loss of context, the item should be made into a stand-alone item: 
How often do you use homework to assess student progress in science? 



Once or twice a week □ 

Once or twice a month □ 

Once per grading period □ 

Once or twice a year □ 

Never or hardly ever □ 
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Teacher Item Only: 

T33-36. In comparison with other fourth grade teachers, how well prepared are you to teach 
each of the following? Fill in one box on each line. 





Excellently 


Very well 


Fairly well 


Poorly 


33. Biology/life science 


□ 


□ 


□ 


□ 


34. Chemistry 


□ 


□ 


□ 


□ 


35. Physics 


□ 


□ 


□ 


□ 


36. Earth science 


□ 


□ 


□ 


□ 



Response Category 


Subject Area 


Biology/ 
life science 


Chemistry 


Physics 


Earth science 


Excellently 








1 


Very well 


1 






1 


Fairly well 


5 


3 


3 


4 


Poorly 




3 


3 





Discussion 

For one teacher, “Excellently prepared” meant “a science teacher — someone trained to teach 
science to fourth graders.” “Very well prepared” meant “someone who studied how to teach fourth 
graders science.” “Fairly well prepared” meant “a general understanding,” and “Poorly prepared” meant 
that “one didn’t know squat.” 

For another teacher, “Excellently prepared” and “Very well prepared” meant “someone who 
majored in science.” “Fairly well prepared” meant “someone who is as qualified as most [other 
teachers],” and “Poorly prepared” meant “someone with very little science background.” 

For a third teacher, “Excellently prepared” meant “someone with college-level training in the 
subject and who has taught it for 5 years and attended workshops.” “Very well prepared” meant 
“someone who might have had more experience but less higher level training.” “Fairly well prepared” 
meant “someone who has high school chemistry and Science on a Shoestring experience,” and “Poorly 
prepared” meant “someone who never had a college level class.” These descriptions seem to be 
consistent. Accordingly, the scale appears to work as intended. 

Recommendations 

None. Modifications do not appear to be necessary. 
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Discussion of Fourth Grade Science Findings 

Discrepancy rates. Discrepancy rates for selected items are presented in table 5 (below). For 
these selected items, the use of a simpler rating scale, using vague quantifiers, was generally associated 
with lower discrepancy rates. Nonetheless, discrepancy rates were quite high. The reasons for these and 
other discrepancies are discussed below. 

Table 5. Discrepancy Rate for Fourth Grade Science Items, by Type of Scale 



When you study science in school, how often do you do each of 
the following: 


4-point scale 
(n=16) 


3-point 
scale (n=7) 


Discuss science in the news 


69% 


43% 


Do hands-on activities in science 


75% 


0% 


Talk about measurements and results from your hands-on 
activities 


75% 


29% 


Use a computer for science 


13% 


57% 


Use library resources for science 


69% 


29% 


When you study science in school, how often does your teacher 
do each of the following? 






Talk to the class about science 


81% 


29% 


Do a science demonstration 


88% 


29% 


Show a science videotape or science television program 


63% 


71% 


Use computers for science (e.g., such as science software, 
telecommunications) 


13% 


33% 



NOTE: The number of students (n) is the modal number of respondents to each item. 



Nature of science instruction. There are several factors which must be considered in the 
development of fourth grade science instructional practice items. The first of these is that science is often 
integrated with the teaching of other activities. For instance, as one teacher pointed out, a social studies 
lesson about the pioneers could include discussions of how pioneers made their own candles and soap. It 
is not difficult to envision how mathematics and reading instruction could also be integrated with science 
instruction. 

Furthermore, science activities are difficult for students to distinguish and label as such. For 
example, lessons on the weather, earthquakes, or space probes might be seen as current events, science, or 
just something the teacher was talking about. For this reason, items that ask, “When you study science in 
school ...” can often lead to underreporting. 

A third factor complicating assessment is that some fourth grade teachers do not teach science. 
They may send their students to another teacher for science lessons. Accordingly, their responses to 
items about teaching science should be linked to the person providing the instruction: “When your 
students are taught science, how often does the teacher ...” 

Time frame issues. A series of items about science activities begins with the stem: “Have you 
ever done hands-on activities or projects in school with any of the following?” This stem evoked reports 
of activities performed in grades other than fourth grade (as well as activities performed outside of 
school). These responses are logical and consistent with alternative interpretations of the stem: 



1) There is no explicit restriction to the fourth grade. 

2) There is no explicit linkage between “hands-on activities” and “in school.” 
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Many behavioral frequency items begin with the phrase, “When you study science in school, how 
often do you do each of the following:” The denominator for responding to these items is explicitly 
defined as “when you study science in school” — that is, the days when you study science in school. 
Accordingly, the response option, “Almost every day,” is intended to mean “Almost every day that 
science is studied.” However, when events, such as science teaching are episodic in nature, rather than 
occurring in a predictable, daily fashion, and when other response options are expressed in terms of per 
week rates (“1-2 times per week”) or per month rates (“1-2 times per month”), the explicit denominator 
for the calculation of rates cannot logically be “days when science is taught.” Instead, the denominator 
for calculating rates consistent with these options is, implicitly, calendar weeks or calendar months. So, if 
science is taught twice a week and the focal behavior occurs in almost every lesson, either “Almost every 
day” or “1-2 times per week” are valid ways of reporting the same behavioral frequencies, depending on 
whether the implicit or the explicit time frame is employed. 

Discrepancies are exacerbated by the fact that teachers generally interpret the item in terms of its 
implicit denominator: “every school day.” 

Comprehension. Many of the words and phrases used in these items were not familiar to all 
fourth graders. Words and phrases that were not consistently interpreted by fourth grade students 
included: 

♦ Hands-on activities 

♦ ...with electricity 

♦ chemicals 

♦ simple machines 

♦ levers 

♦ pulleys 

♦ participation in a science fair, festival, or special science day 

♦ science in the news 

♦ talk about measurement and results 

♦ library resources 

♦ science demonstration 

♦ e.g. 

♦ science software 

♦ telecommunications 

♦ science field trip 

“Participation in a science fair, festival, or special science day” was sometimes interpreted 
merely as attendance at the event. 

Attempts were made to define certain constructs through examples: 

♦ Electricity (for example, batteries and flashlights) 

♦ Chemicals (for example, mixing or dissolving sugar or salt in water) 

Students would frequently respond only to the examples; they were not always able to generalize the 
construct of interest. 
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Fourth grade students have difficulty understanding abstract constructs. “Learning science is 
mostly memorizing” was usually interpreted in ways other than intended by the item writer. 

Teachers were inconsistent in their interpretation of the following phrases: 

♦ library resources 

♦ science field trip 

♦ science demonstration 

Teachers did not know whether to include CD-ROMs and encyclopedias in the classroom as “library 
resources.” They were uncertain whether a field trip with science components should be considered as a 
“science field trip.” Fourth grade field trips frequently provide educational experiences in a number of 
different subjects. Finally, there was no clear agreement about the critical components of a “science 
demonstration.” 

Loss of Context. When items are presented in list format, both students and teachers would 
frequently lose context. They would respond to the listed option as a stand-alone item instead of 
responding with respect to the conditions specified in the stem. When students were asked: 

When you study science in school, how often do you do the following? 

Use a computer for science? 

at least one student responded about use of a computer both inside and outside of school. The context of 
studying science in school was lost. Similarly, when teachers were asked: 

These questions refer to your science instruction in general. About how often do your science 
students do each of the following? 

Use a computer for science? 

at least one teacher lost context and responded about her use of computers for science. And, when a 
teacher was asked: 

How often do you use each of the following to assess student progress in science? 

Homework? 

at least one teacher responded about the frequency with which she assigns homework. 

In another situation, inappropriate maintenance of context was responsible for inappropriate 
responses by nine (of 21) students. The item asked: 



About how much time do you usually spend each day on science homework? 


I don’t have science 


□ 


None 


□ 


Vi hour 


□ 


1 hour 


□ 


2 hours 


□ 


More than 2 hours 


□ 



For these nine students, the concept of science homework was made so salient that they interpreted “I 
don’t have science” as “I don’t have science homework.” Even though they had science (class), they 
selected the option intended to indicate that they weren’t getting any science lessons. 
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“Check all that apply” items. A series of items began with: 

Have you ever done hands-on activities or projects in school with any of the following? 

Several options, each followed by a check box, were listed underneath this stem. When students did not 
understand the options (or the question), students would leave the box unchecked. This makes it 
impossible to distinguish true “No” responses from “Don’t know” responses. 
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Fourth Grade Mathematics Student and Teachers 

Student Item: 

1. When you do mathematics in school, how often do you do each of the following? Fill in one 
box on each line. 

Do mathematics problems from textbooks? 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 

Do mathematics problems from their textbooks? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 68% (13 out of 19 responses) 

Number children reporting more frequently 12 

Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


5 


3 


2 


2 


1-2 times/ month 




ill M 






1-2 times/ week 








5 


Almost every day 


1 






1 
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Discrepancy rate 



33% (2 out of 6 responses) 



Number children reporting more frequently 2 

Number children reporting less frequently 0 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 








1-2 times/ week 




4 


2 


Almost every day 









Discussion 

The discrepancies in this item were mostly due to students overreporting the frequency of use of 
textbooks for doing mathematics problems. As shown above, a total of 14 out of 25 students overreported 
this behavior. Only one student reported this behavior taking place less frequently than the teacher did 
and this was due to a recording error on the part of the student. 

At least four students ignored the words “from textbooks” and thus interpreted the question in a 
more general way: How often do you do math or math problems? This may reflect the lack of a textbook 
or a lack of comprehension of the term textbook. At least two students explicitly did not have a clear 
definition of “textbooks.” One student interpreted textbook as “a book you write math notes in,” like a 
math journal. Another student thought textbook referred to “a class book” — any book you use for math in 
class. 



Three students lost the school context of the question and included the use of math textbooks for 
doing homework. 

The reasons described above all led to students reporting the use of textbooks for doing 
mathematics problems more frequently than their teachers. 



Recommendations 

Students do not appear to have a common understanding of the term “textbook.” Consider 
deleting this term from the question. In order for students not to lose the school context of the question, 
“in class” could be added: 

Do mathematics problems in class? 
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Student Item: 

2. When you do mathematics in school, how often do you do each of the following? Fill in one 
box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Do mathematics problems on worksheets? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



74% (14 out of 19 responses) 
10 
4 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 








4 


1 -2 times/ month 










1-2 times/ week 


3 




2 


6 


Almost every day 


1 






3 
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Discrepancy rate 



17% (1 out of 6 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 








1-2 times/ week 


1 


i 




Almost every day 






4 



Discussion 

A total of 15 out of 25 students reported different frequencies than their teachers in this item. Ten 
students overreported the use of worksheets for doing math problems, and five students underreported this 
behavior. There were several reasons for the discrepancies. 

At least six students did not have a clear definition of “worksheets” which led to both 
underreporting and overreporting of the behavior. Some students interpreted “worksheets” as tests, others 
thought it meant copying down from the teacher on a piece of paper, and one student interpreted the word 
to mean “math folder.” 

Similar to the previous item, four students overreported the use of worksheets because they lost 
the school context of the question and included the use of worksheets for homework. 

Two students included tests they do on worksheets in their responses, which led to overreporting 
of the behavior since their teachers did not include the use of worksheets for this purpose. Thus, these 
students had a more inclusive definition of worksheets than their teachers did. 

Some teachers included homework in their responses and others did not. The wording of the 
teacher item does not convey a clear intent as to whether homework should be included or not. The item 
stem asks, “How often do the students in your class do each of the following?” The phrase, “How often 
do the students in your class” can be interpreted as either asking about “your students” or asking about 
what “students do while in your classroom.” With the first interpretation, homework should be included; 
with the second, homework should be excluded. 



Recommendations 

The term “worksheet” does not have a clear meaning to fourth graders. They either misinterpret 
the term or overreport the frequency with which they use worksheets because they include homework in 
their responses or tests they do on worksheets. The item is a candidate for deletion. 

The intent of the teacher item should be clarified to deal with the issue of homework inclusion. 
That is, is this item intended to deal with activities in the classroom or with activities of the students who 
comprise the class? 
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Student Item: 

3. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Solve mathematics problems with a partner or in small groups? 

Teacher Item: 



How often do the students in your class do each of the following? Fill in one box on each line. 


Solve mathematics problems in small groups or with a partner? 


Original scale (both): 


Revised scale: 




Almost every day 


□ Almost every day 


□ 


Once or twice a week 


□ Sometimes 


□ 


Once or twice a month 


□ Never 


□ 


Never or hardly ever 


□ 





Discrepancy rate 



74% (14 out of 19 responses) 



Number children reporting more frequently 6 

Number children reporting less frequently 8 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 




1 


1 




1-2 times/ month 


2 


2 


4 




1-2 times/ week 


1 




2 




Almost every day 




1 


4 


1 
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Discrepancy rate 



100% (6 out of 6 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 6 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 


vM'% 






1-2 times/ month 








1-2 times/ week 


2 






Almost every day 


1 


3 


* 'Jv 



Discussion 

Regardless of the scale employed, this item had a high error rate. A total of four-fifths of the 
students (20 out of 25) reported solving math problems with a partner or in small groups with different 
frequencies than their teachers. Fourteen students underreported this behavior and six students indicated 
that they engaged in these activities more often than their teachers reported. 

The discrepancies were due largely to the fact that students and teachers had different definitions 
of what “solving math problems with a partner or in a small group” meant. Some students underreported 
the behavior because they defined it as cheating, sharing, or helping each other. Other students 
overreported the behavior because they included informal discussions with their classmates. 



Recommendations 

Fourth graders do not have a common understanding of either “solving math problems” or “with 
a partner or in a small group.” We recommend only asking teachers about this instructional practice. 
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Student Item: 

4. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Work with objects like rulers, counting blocks, or geometric shapes? 

Teacher Items: 

How often do the students in your class do each of the following? Fill in one box on each line. 

Work with objects like rulers? 

Work with counting blocks or geometric shapes? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 



74% (14 out of 19 responses) 



Number children reporting more frequently 9 

Number children reporting less frequently 5 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 


1 


2 


3 


4 


1-2 times/ week 




3 


2 


2 


Almost every day 


1 






1 



BEST COPY AVAILABLE 



0 
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Discrepancy rate 



67% (4 out of 6 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 4 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 








1-2 times/ week 




2 




Almost every day 




4 





Note: Teacher responses to the two listed items were combined into a single teacher response. Two 
of the six teachers provided different responses to the two items (e.g., one teacher responded 
“once or twice a week” to the first item and “once or twice a month” to the second item). In 
these cases, the most frequent response was chosen as the overall teacher response. 

Discussion 

A total of 18 students (or 72%) provided different responses about the use of objects like rulers, 
counting blocks, or geometric shapes than their teachers. There was no systematic bias since nine 
students overreported this behavior and another nine students underreported the behavior. Many different 
problems with this item were detected, as described below. 

Five students underreported the use of rulers, counting blocks, or geometric shapes because they 
only focused on one of the examples (e.g., counting blocks). Thus, the construct was not generalized. 

Five students did not know what “geometric shapes” meant. However, only in two cases did this 
lead to discrepancies where students underreported the behavior. 

Three students lost the context of math, which led to overreporting because they included the 
usage of measuring instruments in other classes or for other purposes (e.g., using a ruler to make a 
drawing). One teacher also lost the math context of the question and included the use of rulers in a 
variety of activities. 

At least two students overestimated the behavior due to time estimation problems. For example, 
one student was not able to average the behavior over the school year and answered with respect to the 
future. 



Recommendations 

Consider changing the items in the following way: 

Student item: Work with rulers , blocks , or shapes during math lessons 

Teacher item: Work with rulers, counting blocks, or geometric shapes in math 

Also, anchoring the time frame (e.g., in the past week) and changing the response options 
accordingly might help alleviate time estimation problems. 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

5. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Write a few sentences about how you solved a mathematics problem? 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 

Write a few sentences about how to solve a mathematics problem? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



61% (1 1 out of 18 responses; 1 missing) 

6 

5 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 










1-2 times/ week 




2 


6 


6 


Almost every day 


2 




1 


1 
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Fourth-Grade Mathematics Student and Teachers 



Discrepancy rate 



67% (4 out of 6 responses) 



Number children reporting more frequently 1 

Number children reporting less frequently 3 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 








1 -2 times/ week 




1 


1 


Almost every day 




3 


1 



Discussion 

At least two students did not understand the question, which led to overreporting of the behavior. 
These students interpreted the question to mean: correcting a problem on the board, doing daily edits, 
writing down what the problem means, or a definition of a word. 

At least two students had trouble estimating time using the old four-point scale, which led to both 
under-and overreporting. Another issue related to time is the problem with estimating behavioral 
frequencies. One student overreported the behavior because he was thinking of the beginning of the year 
when they had done a lot of problem solving and writing in math. Another student underreported because 
the class was doing standardized testing during the week of the interview. These students were not able 
to average the behavior over the course of the school year. 

Two students included writing they do for their math homework, which led to overreporting of 
the behavior. Thus, they lost the context (“in school”) of the question. 

Another student had trouble with the format of the question and lost context completely. She 
forgot the stem of the question and interpreted the question literally - she thought she had to write 
sentences about how she solved a mathematics problem! 



Recommendations 

The concept of “writing a few sentences about how you solved a mathematics problem” is not 
clear to some fourth graders. Consider: 

Answering a math question by writing sentences . 

Also, simplifying the scale or making the item dichotomous should alleviate some of these time 
estimation problems. As mentioned earlier, it is very important that the context of the question be 
provided and maintained. Similar to the previous question, “during math lessons” could be added to the 
question to insure the maintenance of context. 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

6. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Take mathematics tests? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



32% (6 out of 19 responses) 
2 
4 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 




4 


1 




1-2 times/ week 




4 


9 


1 


Almost every day 
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Fourth-Grade Mathematics Student and Teachers 



Discrepancy rate 



33% (2 out of 6 responses) 



Number children reporting more frequently 2 

Number children reporting less frequently 0 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 




4 


2 


1-2 times/ week 








Almost every day 









Discussion 

In spite of the relatively low discrepancy rate, the item was problematic because of the definition 
of “mathematics tests.” What counts as a math test? Some teachers included only monthly tests; other 
teachers included both monthly and weekly tests. Similarly for the students, some students included 
weekly tests (e.g., “times” tests), whereas other students only counted the “big math tests” (i.e., the 
monthly tests). As one student put it “the small tests [i.e., the weekly tests] are not real tests.” In 
addition, one student included standardized testing and overreported because the interview took place 
during a week of testing. These differences in the definition of the construct led to both over- and 
underreporting and accounted for most of the discrepancies. 

There were a few other item problems, as well. At least one student overreported because he was 
not able to average the behavior over time. Three weeks earlier, he had taken math tests every day. One 
student was not able to match her verbal response (one test a week) to the response options and checked 
“once or twice a month.” 



Recommendations 

The term “mathematics tests” is not clear. Should weekly tests (the so-called “times” tests or 
quizzes) be included? And what about standardized testing? Teachers often do not consider these in their 
reporting. Consider: 

Students: Take mathematics tests and quizzes that your teacher scores 
Teachers: Take mathematics tests and quizzes . Do not include standardized tests . 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

7. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Talk to the class about your mathematics work? 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 

Talk to the class about their mathematics work? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



89% (17 out of 19 responses) 

4 

13 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


1 


1 






1-2 times/ month 










1-2 times/ week 


6 




1 


3 


Almost every day 


3 


1 


3 
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Fourth-Grade Mathematics Student and Teachers 



Discrepancy rate 



100% (6 out of 6 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 6 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 
1-2 times/ month 




ISM 




1-2 times/ week 








Almost every day 


3 


3 





Discussion 

This item is subject to individual variation with respect to the focal behavior. That is, 
discrepancies may reflect within class differences. Nonetheless, there are many indications of problems. 

Teachers and students had very different interpretations of what “talk to the class about your 
mathematics work” meant. Three of the six teachers did not perceive much of a difference between this 
item (“Talk to the class about their mathematics work”) and the succeeding item on the teacher 
questionnaire: “Discuss solutions to mathematics problems with other students.” This led to teachers 
having more informal interpretations of the construct, where they would include group projects. In 
contrast, at least nine students had more formal interpretations (e.g., standing in front of the class and talk 
about math), which led to students underreporting the behavior compared to their teachers. 

Another source of discrepancy arose due to teachers reporting about the typical student as 
opposed to the frequency with which any student performed the behavior of interest. Thus, 
underreporting occurred in four cases when students felt that they were not called upon as frequently as 
the typical student. The reverse also happened in one case, where a student overreported the behavior 
because she felt that she spoke more often in front of the class than other students. 

Two students underreported the behavior because they interpreted the item as asking about 
whether they cheat in math. They explained that their teachers do not want them to share and that they 
are not supposed to show what grade they got or how they solve the math problems. 

Two respondents (one student and one teacher) lost context. One student included other subjects 
(e.g., social studies), which led to overreporting. One of the teachers initially responded with respect to 
her own behavior, as opposed to her students’ behavior. 



Recommendations 

Teachers have a hard time determining whether “formal” and/or “informal” presentations should 
be included. If the intent of the item is to get at formal presentations, an item asking about how often 
“ Students speak in front of the whole class about their mathematics work ” could be considered. 

A decision must be made about whether the teacher item is referring to any student or the typical 
student and the stem modified accordingly. 
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Fourth-Grade Mathematics Student and Teachers 



We propose changing the student item to make it parallel to the proposed change to the teacher 
item: “Speak in front of the whole class about your mathematics work. ” 



The Measurement of Instructional Background Indicators 



86 



Fourth-Grade Mathematics Student and Teachers 



Student Item: 

8. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Do 10 or more practice problems in mathematics by yourself? 

Teacher Validation Probe (not a survey item): 

How often do your students do 10 or more practice problems in mathematics, by themselves, in 
class? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



79% (15 out of 19 responses) 
6 
9 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 




2 


1 


1 


1-2 times/ month 






1 


1 


1-2 times/ week 






1 




Almost every day 


1 


1 


7 


3 
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Fourth-Grade Mathematics Student and Teachers 



Discrepancy rate 



50% (3 out of 6 responses) 



Number children reporting more frequently 1 

Number children reporting less frequently 2 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 








1-2 times/ week 






1 


Almost every day 




2 


3 



Discussion 

Response discrepancies were due to item-specific reasons: the definition of “practice problems,” 
the loss of context, and difficulties with rate estimation. 

Students interpreted the term “practice problems” in various idiosyncratic ways, which led to both 
over- and underreporting. Below are some of these interpretations: 

♦ When the teacher puts a problem on the board and we have to figure it out 

♦ [Practice problems] make it easier to do harder problems; practice problems are less difficult 
than the pages in a textbook 

♦ Practice, like when you keep on trying to see if the answer is right 

♦ An exercise that helps you remember things 

♦ Practicing math at home 

In addition, two students included small tests (quizzes) and times tables in their definition of 
“practice problems,” which led to overreporting of the behavior. 

At least five students lost context and included homework or practicing math at home in their 
responses. 

Two students had trouble with recall and time estimation and overreported the frequency. 

Recommendations 

The term “practice problems” is not clear to most fourth graders. The intent of the item should be 
clarified (e.g., how are practice problems different from problems on worksheets?) and the item should be 
either revised accordingly or deleted. 

As mentioned earlier, maintaining the context of the item (in this case, the context of math 
behaviors in the classroom) is very important. 

Simplifying the scale will alleviate some of these problems. 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

9. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Discuss solutions to mathematics problems with other students? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



58% (1 1 out of 19 responses) 
0 
11 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


2 








1-2 times/ month 










1-2 times/ week 




1 


5 




Almost every day 


7 




3 


1 
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Fourth-Grade Mathematics Student and Teachers 



Discrepancy rate 



100% (6 out of 6 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 6 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 


HU 11 fill 






1-2 times/ month 




|pp£ ? , ; % • -|i|| 




1-2 times/ week 




w | s : | 

4*11 ' 4fet 




Almost every day 




6 





Discussion 

The term “discuss solutions to mathematics problems with other students” is unclear to both 
teachers and students. 

As mentioned earlier, three of the six teachers did not perceive much of a difference between this 
item (“Discuss solutions to mathematics problems with other students”) and the preceding item on the 
teacher questionnaire: “Talk to the class about their mathematics work.” 

Four students explicitly interpreted the item as asking about cheating and therefore answered 
“never or hardly ever.” As one student put it: “You should not talk to other students about your answers 
because if you tell them, they will never learn!” Other students had more formal or idiosyncratic 
interpretations of the question than their teachers, which led to underreporting, as well. In addition, two 
students thought that this item and “Solve mathematics problems with a partner or in small groups” were 
similar. 



Two respondents (one student and one teacher) lost context. One student included homework in 
the answer and one of the teachers initially responded with respect to her own behavior, as opposed to her 
students’ behavior. 

Two students could not read the word “solutions.” 

Recommendations 

If the intent of the item is to get at informal discussions, an item asking about how often 
“Students talk with other students about how they solve a math problem” could be considered. 

Fourth graders’ literal interpretation of this item, as with “Solve mathematics problems with a 
partner or in small groups,” makes it hard to ask about these behaviors without some students interpreting 
them as cheating. Consider not asking fourth graders about these types of instructional behaviors. 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

10. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Use a computer? 



Original scale (both): 

Almost every day 
Once or twice a week 
Once or twice a month 
Never or hardly ever 



Revised scale: 

□ Almost every day □ 

□ Sometimes □ 

□ Never □ 

□ 



Discrepancy rate 37% (7 out of 19 responses) 

Number children reporting more frequently 5 

Number children reporting less frequently 2 



Teacher 


Student Responses 


Responses 


Never or hardly ever 


1-2 times/ month 


1-2 times/ week 


Almost every day 


Never or hardly ever 


8 


1 




2 


1-2 times/ month 


2 


4 


1 


1 


1-2 times/ week 










Almost every day 
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Fourth-Grade Mathematics Student and Teachers 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



0% (0 out of 6 responses) 
0 
0 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 


6 






1-2 times/ month 








1-2 times/ week 








Almost every day 









Discussion 

In spite of the relatively low discrepancy rate, the item had one main problem. At least four 
students lost the context of the item and reported use of a computer anywhere and for any purpose. One 
teacher also lost context initially and included students’ computer usage in other subjects than math. 

One student underreported because he did not include math computer games in his response. 



Recommendations 

To avoid the loss of context, we propose the following revisions: 

Student item: Use a computer for math work in school 

Teacher item: Use a computer for math 

We assume that math computer games should be included. If not, this should be clarified in the 

item. 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

1 1. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Use a calculator? 

Teacher Validation Probe (not a survey item): 

About how often do your students use a calculator to do their math class work? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



42% (8 out of 19 responses) 

5 

3 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


11 


2 


2 


1 


1-2 times/ month 


3 








1-2 times/ week 










Almost every day 
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Fourth-Grade Mathematics Student and Teachers 



Discrepancy rate 



66% (4 out of 6 responses) 



Number children reporting more frequently 2 

Number children reporting less frequently 2 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 




2 




1-2 times/ month 


2 


2 




1-2 times/ week 








Almost every day 









Discussion 

Three students overreported their usage of calculators because they included the Dollar Word 
Game that they do on their calculators. (The Dollar Word Game is a calculator game, where each letter is 
assigned a value corresponding to the number of the letter in the alphabet, [e.g., E=5, Z=26]. The object 
is to find words for which the point values of the letters sum to 100.) Their teacher did not count this 
game, since it is not part of math lessons. 

At least two students had trouble averaging the use of calculators over the course of the school 
year. Their teacher reported that her students had used calculators a lot in the beginning of the school 
year, but some students were not able to include this usage in their estimates which led to underreporting. 

One student lost context and included calculator usage both at school and at home. 

One student had a different math teacher in the beginning of the school year. He included 
calculator usage in this class in his response, which led to apparent overreporting. 



Recommendations 

In a multiple subject classroom, instruction in mathematics can occur in the context of other 
subjects. 

As mentioned earlier, anchoring the time frame (e.g., in the past week) and changing the response 
options accordingly would help alleviate these time estimation problems. 

To avoid the loss of context, we propose the following revision: 

Student item: Use a calculator for math in school 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

12. Do you have a calculator that you can use to do mathematics schoolwork? 

Teacher Validation Probes (not survey items): 

Do all of your students have calculators? 

Do you provide calculators to students who can not afford them? 

Both: 



Yes □ 
No □ 



Discrepancy rate 



36% (9 out of 25 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 


5 


3 


Yes 


6 


11 



Discussion 

In spite of the relatively low discrepancy rate, there were problems with this item. At least three 
students interpreted the term “can use” as “are allowed to use,” which led to students responding “no.” 
Thus, even though students have access to calculators they respond “no” to the item because they are not 
allowed to use them for math. 

Similar to the item problem described above, two students who have access to calculators 
responded “no” to the item because they do not use them in their math class. Conversely, one student 
correctly responded “yes” because she owns a calculator. However, her teacher’s response was “no” 
because not all of the students in her class have access to calculators. This student was not one of them. 



Recommendations 

We proposed the following revision: 

Do you have a calculator, either at school or at home, that you can for when you want? 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

13. For mathematics class, how often do you use a calculator for each of the following activities? 
Fill in one box on each line. 

Classwork? 

Teacher Validation Probe (not a survey item): 

About how often do your students use a calculator to do their math class work? 



Original scale (both): 

Almost every day 
Once or twice a week 
Once or twice a month 
Never or hardly ever 





Revised scale: 




□ 


Almost every day 


□ 


□ 


Sometimes 


□ 


□ 


Never 


□ 


□ 







Discrepancy rate 



26% (5 out of 19 responses) 



Number children reporting more frequently 5 

Number children reporting less frequently 0 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


14 


2 


2 


1 


1-2 times/ month 










1-2 times/ week 










Almost every day 












The Measurement of Instructional Background Indicators 



96 



Fourth-Grade Mathematics Student and Teachers 



Discrepancy rate 



67% (4 out of 6 responses) 



Number children reporting more frequently 4 

Number children reporting less frequently 0 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 


2 


4 




1-2 times/ month 








1 -2 times/ week 








Almost every day 









Discussion 

A total of 9 students (or 36%) overreported the use of calculator for math classwork for various 
idiosyncratic reasons. Two students lost the context of the question. One of these students interpreted the 
question as: “How often do you do classwork ?” Thus, the student forgot both the math and the calculator 
contexts of the question. The other student included calculator usage in other subjects. 

One student included classwork done at home. That is, the student interpreted classwork as work 
that could be done both in class and at home. 

Another student interpreted “never or hardly ever” as “never in my whole year.” Since she had 
used a calculator in January (a couple of months earlier), she said “once or twice a month” — the closest 
option to “never or hardly ever.” 

One student included the Dollar Word Game mentioned earlier. 

One student had a different math teacher in the beginning of the school year. He included 
calculator usage in this class in his response, which led to apparent overreporting. 

One student chose “once or twice a month” because “when the teacher got no sleep or is nice, she 
allows us to use them [the calculators]!” 



Recommendations 

To deal with loss of context, the following revision should be considered: 
How often do you use a calculator during math lessons? 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

14. For mathematics class, how often do you use a calculator for each of the following activities? 
Fill in one box on each line. 



Homework? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Teacher Validation Probe (not a survey item): 

Do your students use calculators for their math homework? 



Yes □ 
No □ 



Discrepancy rate 



80% (4 out of 5 responses; 14 missing) 



Teacher 

Responses 


Student Responses 


Never or 
hardly ever 


Yes 


1-2 times/ 
month 


1-2 times/ 
week 


Almost 
every day 


No 


1 


1 


Yes 


3 





Discrepancy rate 



50% (3 out of 6 responses) 



Teacher 

Responses 


Student Responses 


Never 


Yes 


Almost 

Sometimes every day 


No 






Yes 


3 


3 
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Fourth-Grade Mathematics Student and Teachers 



Discussion 

Three teachers indicated that they did not know whether their students used calculators to do their 
math homework. Thus, it was not possible to validate the item for these teachers’ students, which is the 
reason that information is missing for 14 of the 25 students. In fact, the one case where the student 
indicated that she uses a calculator for her math homework and the teacher said “no” is an example of the 
teacher simply not knowing what students do at home. 

In another case, a student lost context interpreting the question as: How often do you do math 
homework? 

Some of the students who said that they never used a calculator for their math homework even 
though their teachers said they probably did, provided explanations such as: 

♦ The teacher doesn’t want students to get addicted to using calculators because they give you 
the answers. 

♦ Mom won’t let me. 

Recommendations 

In order for students not to lose context, the item should be revised in the following way: 

How often do you use a calculator for your math homework? 
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Fourth-Grade Mathematics Student and Teachers 



Student Item: 

15. For mathematics class, how often do you use a calculator for each of the following activities? 
Fill in one box on each line. 



Tests or quizzes? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Teacher Validation Probes (not survey items): 



Are students ever allowed to use calculators for math tests? 

What are your rules about the use of calculators on math tests? 

Can they use them for all math tests? 

Yes □ 

No □ 



Discrepancy rate 21% (4 out of 19 responses) 



Teacher 

Responses 


Student Responses 


Never or 
hardly ever 


Yes 


1 -2 times/ 
month 


1 -2 times/ 
week 


Almost 
every day 


No 


15 


2 


Yes 


2 





Discrepancy rate 83% (5 out of 6 responses) 



Teacher 

Responses 


Student Responses 


Never 


Yes 


Sometimes 


Almost 
every day 


No 


1 


i 


Yes 


4 
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Fourth-Grade Mathematics Student and Teachers 



Discussion 

Only one teacher indicated that her students are allowed to use calculators for math tests. 
However, she also indicated that they are only allowed on certain types of tests with harder problems. All 
of her six students responded that they are not allowed to use calculators for tests or quizzes. As some of 
the students explained: 

♦ We would get busted and would get a detention 

♦ The teacher tells us to clear our desks of everything - it is a rule and there are no exceptions 

Another problem was related to interpretation of the term, “Tests or quizzes.” The teacher and 
students were thinking of different tests (e.g., teachers did not include standardized tests). 

At least two students lost the math and calculator context of the question. Thus, they interpreted 
the question as: How often do you have tests or quizzes? 

One student did not know what “quizzes” meant. 



Recommendations 

In order to deal with the item problems described above, we propose revising the item as follows: 
How often do you use a calculator for math tests? 
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Student Item: 

16. About how much time do you usually spend each day on mathematics homework? 



I am not taking mathematics this year □ 

None □ 

15 minutes □ 

30 minutes □ 

45 minutes □ 

1 hour □ 

More than 1 hour □ 



Teacher Item: 



Approximately how much mathematics homework do you assign to students in this class each 
day? 


None 


□ 


15 minutes 


□ 


30 minutes 


□ 


45 minutes 


□ 


One hour 


□ 


More than one hour 


□ 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



61% (14 out of 23 responses) (2 missing) 
10 
4 



Teacher 

Responses 


Student Responses 


1 am not 
taking math, 
this year 


None 


15 

minutes 


30 

minutes 


45 

minutes 


1 hour 


More than 1 
hour 


I am not taking 
math, this year 
















None 
















15 minutes 






7 


6 


2 


1 


1 


30 minutes 






4 


2 








45 minutes 
















1 hour 
















More than 1 
hour 
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Discussion 

One teacher does not assign math homework. If students do not finish their math work at school, 
they are supposed to complete it at home. So, the time spent on math at home will vary from student to 
student. For this reason, the teacher did not respond to the question and validation information is 
therefore missing for her two students. 

One teacher initially responded “30 minutes,” since she gives two math homework assignments a 
week, of about 30-40 minutes duration. She then adjusted her response and answered in terms of “each 
day.” Decomposition of the item might be considered (e.g., how many days a week do you assign math 
homework; how long does the typical assignment take). Or, the timeframe might be shifted to ask about 
the typical week. Furthermore, when students respond to an item like this -- asking about an event that 
occurs fairly frequently, but not daily, they will often respond with respect to the typical amount of time 
in which they engage in this activity. That is, if math homework is given three times a week and it 
usually takes 15 minutes to complete, they will respond 15 minutes per day. However, the item writer 
probably intended that the answer reflect the total amount of time per week, divided by the number of 
days in a week. 

Two students ignored the word “homework” in the question, which led to overreporting because 
they answered with respect to how much time they spent on math each day. Another student ignored the 
word “mathematics” and answered with respect to time spent on all homework. 

Two students did not have a good sense of time and overreported the amount of time they spent 
doing math homework every day. 

Two students underreported because they are above average in math. They are good at math and 
felt that their math homework was easy. 

One student included time spent on math homework at school and at home. This could be 
considered overreporting. 

One student used contextual information from the scale to respond to the question. He is good at 
math and spends less time than other students. So, he picked the lowest non-zero option on the scale (15 
minutes). 

One student felt that the word “usually” could be taken out. Another student thought the boxes 
were too close together. 

Recommendations 

Decomposition of the item might be considered (e.g., how many days a week do you assign math 
homework; how long does the typical assignment take). Or, the timeframe might be shifted to ask about 
the typical week. 

To deal with some of the issues described above, we propose the following revised wording: 



About how long does it usually take to do your math homework? 



More than one hour 



I don y t have math homework 



15 minutes 
30 minutes 
45 minutes 
1 hour 



□ 

□ 

□ 

□ 

□ 

□ 
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Given that none of the fourth graders responded “I’m not taking mathematics this year” and that 
this option does not apply to fourth graders, we propose deleting it. 

We also propose adding another question to get a sense of the frequency of math homework: 

How often does your teacher give you math homework? 



Every day □ 

Three or four times a week □ 

One or two times a week □ 

Never or almost never □ 
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Student Item Only: 

How much do you agree with each of the following statements? Fill in one box on each line. 

Agree Undecided/Not sure Disagree 

17. I am good at mathematics □ □ □ 



How much do you agree with each of the following statements? Fill in one box on each line. 

Agree Undecided Disagree 

17. I am good at mathematics (2 missing) 8 3 0 

(revised) How much do you agree with each of the following statements? Fill in one box on each line. 

Agree Not sure Disagree 

17. I am good at mathematics (1 missing) 6 5 0 



Discussion 

Seven of the first 13 fourth graders who were interviewed had trouble with the word “undecided” 
in the scale. They either could not read it or did not understand what it meant. Below are some of their 
definitions of the word. 

♦ lam alright at math (as opposed to good or bad at math) 

♦ Not very good at it sometimes 

♦ The middle of something 

♦ Not that good (agree = being smart, undecided = not that good, disagree = not being good) 

In addition, one student, who did not know what undecided meant, chose that option because he 
thought the answer choices were a scale, and he considered himself in the middle. 

Given the problems with the word “undecided,” it was changed to “not sure” half way through 
the interviews. This revision seemed to work much better. 

The item seemed to work as intended. However, one student chose “not sure” because she did 
not “want to brag.” Her dad told her that girls were better at reading and boys were better at math. She 
thought that “not sure” meant “sort of good, sort of bad.” It might be advisable to provide answer choices 
that describe a range of math abilities. Some students pick “not sure” because they want to say that they 
have moderate math abilities. Furthermore, one student included spelling in his answer to Question 17. 
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Recommendations 

Using “not sure” instead of “undecided” eliminated the “undecided” comprehension problem and 
is an improvement. 

If finer self-assessment descriptions are desired, consider changing the question to ask explicitly 
about math abilities if that is the intent of the question. 

How good are you at math? Are you: 

Very good □ 

Good □ 

Okay □ 

Bad □ 
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Student Item Only: 

How much do you agree with each of the following statements? Fill in one box on each line. 

Agree Undecided Disagree 

18. Learning mathematics is mostly memorizing facts □ □ □ 



How much do you agree with each of the following statements? Fill in one box on each line. 



Agree 


Undecided 


Disagree 


18. Learning mathematics is mostly memorizing facts 6 

(1 missing) 


4 


2 



(revised) How much do you agree with each of the following statements? Fill in one box on each line. 



Agree 


Not sure 


Disagree 


18. Learning mathematics is mostly memorizing facts 6 

(1 missing) 


5 


0 



Discussion 

Seven of the 25 students (or 28%) did not understand this question. For example, two students 
chose “not sure” because they did not understand what it meant. The term “memorizing facts” appeared 
to create particular problems. A couple of students had the following definitions: 

♦ I don’t know that much factors 

♦ Do you do multiplication of factors 

However, one student appeared to have a clear definition of what “memorizing facts” meant. She 
defined it as: “A day before a test, you need to get them into your brain (for example, 8x8=64).” 

As previously noted, “undecided” is not a term that is familiar to all fourth graders. 



Recommendations 

Several fourth graders do not understand this question, in particular the term “memorizing facts.” 
Consider deleting it. 
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Student Item: 

19. This year in school, how often have you taken mathematics tests where you were asked to 
provide detailed solutions to problems you had not worked on before? 

Teacher Validation Probes (not survey items): 

How frequently do you ask your students to provide detailed solutions to problems on which they 
had not worked before? 



Original scale (both): 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



88% (14 out of 16 responses; 3 missing) 
14 
0 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ year 


1-2 times/ 
month 


At least once a 
week 


Never or hardly ever 


1 




4 


8 


1-2 times/ year 










1-2 times/ month 






1 


2 


At least once a week 
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Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



60% (3 out of 5 responses; 1 missing) 
3 
0 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


At least once a 
week 


Never or hardly ever 


1 


1 


1 


1-2 times/ year 








1-2 times/ month 




1 


1 


At least once a week 









Discussion 

This question is too long and complicated for fourth graders, as indicated by the high discrepancy 
rates. Most students had to repeat some or all of the question. 



Most of the students (at least 16) misunderstood the question. This often led to overreporting. 
They specifically had trouble with the term “provide detailed solutions,” which many of them chose to 
ignore. Six students interpreted the question as: “How often do you take math tests?” Other 
interpretations included: 

♦ Do you do math tests more than once a week or have you done any at all this year? 

♦ How many times do you have to use details (to solve problems)? 

♦ How often do you have math problems that you have to explain your answers? 

♦ How many tests have you taken in mathematics? 

♦ Do you know how to solve math problems like multiplication, fractions, etc.? 

Two students left the question blank because they did not understand it. 

Two students got the right answer for the wrong reasons. That is, they misunderstood the 
question but still provided a response that was in agreement with their teacher. These students had the 
following interpretations of the question: 

♦ Telling answers out - talking in front of the class. He never talks in front of the class to tell 
answers to a test. 

♦ Writing a story, where there are plots in detail. Solution is the ending of the story. 

One of the students who interpreted the question as: How often do you take math tests? may have 
been prompted by the term “detailed solutions” to answer the way he did (“at least once a week”). He did 
a calculation: he takes two math tests per week but they have only had half a year of math, so that comes 
to one math test per week. 

Two students did not respond to the question because they ran out of time. 
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Recommendations 

This question should not be asked of fourth graders. 
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Teacher Item Only: 



Counting this year, how many years in total have you taught the following subjects? (Include any 
full-time or part-time assignments, but not substitute assignments.) 





2 years 


3-5 


6-10 


11-24 


25 Years 




or Less 


Years 


Years 


Years 


or More 


1. Mathematics 


□ 


□ 


□ 


□ 


□ 



Counting this year, how many years in total have you taught the following subjects? (Include any 
full-time or part-time assignments, but not substitute assignments.) 





2 years 


3-5 


6-10 


11-24 


25 Years 




or Less 


Years 


Years 


Years 


or More 


1. Mathematics 


1 


1 


1 


2 


1 



Discussion 

One teacher included home economics as part of science but not as part of math, even though the 
students use fractions (e.g., half cups, a third of a cup) as part of their cooking instruction. Accordingly, 
she recorded less math than science teaching experience. However, overall the item appeared to work as 
intended. 



Recommendations 

None. 
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Teacher Item Only: 

3-6. How often do you use computers for MATHEMATICS instruction for: 





Almost 
every day 


Once or twice 
a week 


Once or twice 
a month 


Never or 
hardly ever 


Drill and practice 


□ 


□ 


□ 


□ 


Demonstration of new topics in 
mathematics 


□ 


□ 


□ 


□ 


Playing mathematical/learning games 


□ 


□ 


□ 


□ 


Simulations and applications 


□ 


□ 


□ 


□ 


3-6. How often do you use computers for MATHEMATICS instruction for: 






Almost 
every day 


Once or twice 
a week 


Once or twice 
a month 


Never or 
hardly ever 


Drill and practice 


0 


1 


1 


4 


Demonstration of new topics in 
mathematics 


0 


0 


0 


6 


Playing mathematical/learning games 


0 


2 


2 


2 


Simulations and applications 


0 


0 


0 


6 



Discussion 

Two teachers had trouble distinguishing between “drill and practice” and “playing 
mathematical/leaming games,” since many math games are drill and practice exercises. This led to 
double counting. 

Some teachers responded to Questions 3 and 5 with respect to any students’ computer usage, 
whereas other teachers responded with respect to the typical student. 

One teacher did not know what using computers for simulations and applications meant. This is 
probably why she answered “Never or hardly ever.” 



The Measurement of Instructional Background Indicators 



112 



12 0 



Fourth-Grade Mathematics Student and Teachers 



Recommendations 

Consider deleting “playing mathematical/leaming games” since they are invariably “drill and 
practice” exercises. 

It should be specified whether the teacher is supposed to respond with respect to any student or 
with respect to the typical student. 

Providing examples of “simulations and applications” would help teachers understand the 
question better. 
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Discussion of Fourth Grade Mathematics Findings 

Discrepancy rates. Discrepancy rates for selected items are presented in table 6 (below). For 
these selected items, the use of a three-point rating scale, using vague quantifiers, did not have consistent 
impacts on discrepancy rates. Overall, discrepancy rates were quite high — for some items, levels of 
agreement were less than would have been expected by random guessing. In addition to lacking the 
cognitive skills to accurately answer these items, other factors also contributed to the discrepancies 
discussed in this section. 



Table 6. Discrepancy Rate for Fourth Grade Mathematics Items, by Type of Scale 



When you do mathematics in school, how often do you do each of 
the following: 


4-point 

scale 

( n =19) 


3-point 
scale (n=6) 


Do mathematics problems from textbooks 


68% 


33% 


Do mathematics problems on worksheets 


74% 


17% 


Solve mathematics problems with a partner or in small groups 


74% 


100% 


Work with objects like rulers, counting blocks, or geometric shapes 


74% 


67% 


Write a few sentences about how you solved a mathematics problem 


61% 


67% 


Take mathematics tests 


32% 


33% 


Talk to the class about your mathematics work 


89% 


100% 


Do 10 or more practice problems in mathematics by yourself 


79% 


50% 


Discuss solutions to mathematics problems with other students 


58% 


100% 


Use a computer 


37% 


0% 


Use a calculator 


42% 


66% 








This year in school, how often have you taken mathematics tests where 
you were asked to provide detailed solutions to problems you had not 
worked on before? 


88% 


60% 



NOTE: The number of students (n) is the modal number of respondents to each item. 



Nature of mathematics instruction. In the fourth grade, mathematics instruction is generally 
restricted to specific mathematics lessons. However, there are certain instances of mathematics activities 
occurring outside of these lessons. For example, three students overreported their usage of calculators 
because they included the Dollar Word Game that they do on their calculators. (The Dollar Word Game 
is a calculator game, where each letter is assigned a value corresponding to the number of the letter in the 
alphabet, [e.g., E=5, Z=26]. The object is to find words for which the numeric value of the letters sum to 
100.) Their teacher did not count this game in her reports of math instruction, since it is not part of math 
lessons. 



Comprehension. Many of the words and phrases used in these items were not familiar to all 
fourth graders. Words and phrases that were inconsistently interpreted by fourth grade students included: 

♦ textbooks 

♦ worksheets 

♦ solving mathematics problems with a partner or in small groups 

♦ geometric shapes 

♦ write a few sentences about how you solved a mathematics problem 
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♦ mathematics tests 

♦ talk to the class about your mathematics work 

♦ practice problems 

♦ solutions 

♦ discuss solutions to mathematics problems with other students 

♦ undecided 

“Mathematics tests” was problematic because of issues associated with quizzes and standardized 
tests. Teachers would generally exclude both; students would sometimes include the former and usually 
include the latter. “Talk to the class about your mathematics work” generally evoked images of formal 
presentations in the minds of students. However, teachers would also include informal discussions. 
“Discuss solutions to mathematics problems with other students” was interpreted as cheating by at least 
four students! Accordingly, they reported low frequencies of this behavior. 

Fourth grade students also had problems understanding abstract questions, such as, “Learning 
mathematics is mostly memorizing facts.” Over one-quarter (28 percent) of the fourth graders could not 
explain what this statement meant. 

Complex sentences with numerous clauses modifying the construct of interest were nearly 
incomprehensible to fourth grade students. There is no evidence that any fourth grader understood the 
following question: 

This year in school, how often have you taken mathematics tests where you were asked to 
provide detailed solutions to problems you had not worked on before? 

Six (of 19) of the students interpreted this item as asking, “How often do you take math tests?” 
The two students whose responses to this item were congruent with their teachers’ responses (when a 4- 
point rating scale was employed) misinterpreted the item. Agreement was coincidental. 

Attempts were made to define certain constructs through examples: 

When you do mathematics in school , how often do you do the following? 

Work with objects like rulers, counting blocks, or geometric shapes . 

Fourth grade students were not always able to generalize the intended construct. At least five students 
only responded to rulers. 

Loss of context. When items are presented in list format, both students and teachers would 
frequently lose context. They would respond to the listed option as a stand-alone item instead of 
responding with respect to all of the conditions specified in the stem. When students were asked: 

When you do mathematics in school, how often do you do the following? 

Work with objects like rulers ; counting blocks , or geometric shapes? 

Use a computer? 

Three students responded about the use of rulers for non-mathematical purposes and at least four students 
and one teacher responded about computer usage for mathematical and non-mathematical purposes. The 
linkage to mathematics in school was lost. 

Similarly, when teachers were asked: 

How often do the students in your class do each of the following? 
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Talk to the class about their mathematics work? 

Discuss solutions to mathematics problems with other students? 

at least one teacher lost context and responded about her own behavior rather than her students’ behavior. 

Similar loss of context occurred for at least five other items on the fourth grade mathematics 
student and teacher surveys. 
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Fourth Grade Reading Students and Teachers 

Student Item Only: 

1. During the past month, how many books have you read on your own outside of school? 



None 


□ 


lor 2 


□ 


3 or 4 


□ 


5 or more 


□ 



Student Responses 


Frequency 


None 


2 


1 or 2 


8 


3 or 4 


7 


5 or more 


6 



Discussion 

Even though data from teachers could not be used to validate this item, several problems were 
detected. The first problem was associated with the specified timeframe: “during the past month.” Two 
students only thought of the last week. One student ignored “month” and answered with respect to the 
entire “past” (5 books or more). Another student ignored the timeframe altogether and responded with 
respect to the recent past. The last three students defined “during the past month” in the following ways: 

♦ The month of April (the interview took place on April 16*) [as opposed to during the past 30 
days] 

♦ Like during the month or the past month before that 

♦ Before the month maybe; each day; probably next week 

Another problem arose when a student attended to only two of the three conditions defining the 
type of reading of interest. The student attended to the timeframe and the “on your own” conditions but 
included silent reading she does at school, ignoring the “outside of school” part of the question. 

A third problem was associated with different interpretations of “read on your own.” “Read on 
your own” has two distinctly different interpretations: (1) Read by yourself without help from anyone or 
(2) Read on your own volition - that is, read because you wanted to read it. Three students counted 
reading they do at home as part of their reading assignments (e.g., 30 minutes of daily reading). 

An ESL student had a very literal interpretation of the question. He reads books in Spanish and 
English regularly but answered “none” because he “doesn’t go outside to read!” 

One student’s answer was influenced by the response options. He thinks he reads a lot so he 
checked the largest number (“5 or more”). We are unsure if this heuristic produced a correct response. 
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Recommendations 

The timeframe (“during the past month”) is difficult for fourth graders to conceptualize. We 
would propose changing it to “during this school year” - and just asking if students do this. Since the 
intent of the item seems to be reading for fun, the phrase “outside of school” may be confusing. We 
propose consideration of the following: 

During this school year, did you read any books for fun? Do not count any books you read 
for school 

Yes □ 

No □ 
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Student Item Only: 

2. How often do you do each of the following? Fill in one box on each line. 



Read for fun on your own time? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Original Scale 


Revised Scale 


Student Responses 


Frequency 


Student Responses 


Frequency 


Almost every day 


8 


Almost every day 


1 


Once or twice a week 


1 


Sometimes 


6 


Once or twice a month 


4 


Never 




Never or hardly ever 


3 







Discussion 

Several students seemed to understand what was meant by “reading for fun on your own time.” 
One student did not include the chapter books (her daily 30-minute reading assignment) reported in the 
previous question because these were assignments and she “never or hardly ever” reads for fun on her 
own time. Another student distinguished between chapter books for fun and chapter books for 
homework. He only included chapter books for fun in this question. A third student was only thinking of 
school time reading “because sometimes the teacher lets us read the joke books and silent read.” 

Two students had very literal interpretations of what “reading for fun” meant: 

♦ Enjoying the book you are reading 

♦ If the book is fun then reading is for fun 

These students would include assigned reading as “reading for fun.” 

One student had trouble estimating behavioral frequency and another student did not understand 
the format of the question without help. 
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Recommendations 

This item appears to be intended to capture the presence of volitional reading - that is, 
how often do students read because they want to read (in preference to another activity). If this 
is the intent, the item can be revised: 

In addition to reading for schoolwork and homework, how often do you read books, magazines, 
or stories or articles on the computer? 

Almost every day □ 

Sometimes □ 

Never □ 
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Student Item Only: 

3. How often do you do each of the following? Fill in one box on each line. 

Talk with your family or friends about something you have read? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Original Scale 


Revised Scale 


Student Responses 


Frequency 


Student Responses 


Frequency 


Almost every day 


5 


Almost every day 




Once or twice a week 


4 


Sometimes 


5 


Once or twice a month 


1 


Never 


2 


Never or hardly ever 


5 







Discussion 

Two students did not understand the question because of the formatting and might have lost 
context. One of these students read the behavioral description as a command and asked: “Is it telling me 
to do something?” Two other students (one of them an ESL student) thought the question was asking: 
“How often do you read aloud to your family or friends?” 

Two students had trouble with time. One student gave an example from kindergarten, which 
indicated that he had trouble with an unanchored timeframe. Another student understood almost every 
day as “almost every time,” since he tells his friends at recess about the books he reads almost every time 
he reads a new one. 



Recommendations 

In order for students not to lose the context of the question and avoid rate estimation problems, 
we propose the following revision: 

In the past week, did you talk with your family or friends about something you had read? 
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Student Item Only: 

4. How often do you do each of the following? Fill in one box on each line. 



Read a story or a novel? (Version 1-- 13 students) 

Read a story or a book with chapters? (Version 2 — 10 students) 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Original Scale 


Revised Scale 


Student Responses 


Frequency 


Student Responses 


Frequency 


Almost every day 


6 


Almost every day 


5 


Once or twice a week 


5 


Sometimes 


2 


Once or twice a month 


2 


Never 




Never or hardly ever 


2 







Discussion 

The first 13 students were administered this version of the question: 

Read a story or a novel? 

However, it became clear that “novel” is not a fourth-grade word. Eight of the 13 students did not know 
what it meant. Here are some of their definitions of novel: 

♦ A really long book - a novel doesn’t have chapters 

♦ Fairytales or something 

♦ A chapter or a book that’s long 

Two students who could not read the word “novel” answered with respect to stories only. Another 
student responded “never or hardly ever” because he did not know what “novel” meant. However, when 
told that novel is a book, he said he would have answered “almost every day.” 

Due to these interpretation problems, the question was changed to: 

Read a story or a book with chapters? 

After this revision, one student had a literal interpretation of “books with chapters.” She said: “I 
don’t read a book with chapters every day because I can’t finish it.” She did read another book but she 
did not include that because it was not a book with chapters. However, the second version of the question 
seemed to work better than the first version. 
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Three students included stories read both in school and outside of school, whereas one student 
responded “never or hardly every” because “I never read stories unless it is at school.” The question does 
not specify whether stories or books read in school should be included. 

Two students reported about non-fiction reading. One student changed her answer from “once or 
twice a week” to “almost every day” after thinking of history, science, and social studies books. In fact, 
the question does not specify which books to include. 



Recommendations 

“Novel” is not a fourth grade word and should be replaced with “book,” “chapter books,” or 
“books with chapters.” 

The intent of the item has to be clarified before alternatives can be proposed. Should books read 
in school be included? Should non-fiction books or stories read for subjects other than English be 
included? Assuming the intent is to find out about the frequency with which books are read, both in class 
and at home, the following revision should be considered: 

Read a story or read from a book with chapters 
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Student Item: 

5. When you have reading assignments in school, how often does your teacher do each of the 
following? Fill in one box on each line. 

Ask you to do a group activity or project about what you have read? 

Teacher Item: 

How often do you do the following things as a part of reading instruction with this class? Fill in 
one box on each line. 

Ask students to do a group activity or project about what they have read? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



50% (7 out of 14 responses; 2 missing) 
5 
2 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


3 








1-2 times/ month 


1 


3 


2 


1 


1-2 times/ week 


1 




1 


2 


Almost every day 
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Discrepancy rate 



57% (4 out of 7 responses) 



Number children reporting more frequently 2 

Number children reporting less frequently 2 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 


liiBi 






1-2 times/ month 




mum 


1 


1-2 times/ week 


2 




1 


Almost every day 






1 'S-liM 



Discussion 

At least five students misunderstood the question, which led to discrepancies with their teachers’ 
responses. One student read “group activity” as “graph activity,” which led to underreporting. Another 
student defined group activity as an activity with the whole class, which led her to overreport the 
behavior. 

Three students overreported the behavior because they had a more inclusive definition of “group 
activity or project” than their teachers. One student defined these activities as “looking up words together 
or answering question about their book, which they do because they sit at tables arranged in horse shoes.” 
Another student defined it as “a little group with other kids sharing your books - one reads a line or 
paragraph, then another student.” 

Three students included book reports in their responses. As one student said: “Each time we read 
a book, we have to write about it.” There is some textual support for this interpretation, since “or projects 
about what you have read” can be understood as a separate clause, in distinction to a “group activity.” 
However, this interpretation only led to one discrepancy. 

One student’s average was biased due to use of the “availability heuristic” (Tversky). That is, the 
class had just started engaging in group activities in reading, so the behavior was very salient in the 
student’s mind. This saliency led to overreporting. 

One of the teachers did not know what was meant by “group activity or project” and initially lost 
the reading context of the question. 



Recommendations 

Fourth graders do not have consistent interpretations of “group activities or projects” in reading. 
If the question is administered to fourth graders, it could be slightly clarified by adding “group” to 
“projects” — “group activities or group projects,” in order to avoid students including book reports in their 
responses. However, this will not overcome the poor behavioral frequency estimation abilities of fourth 
graders nor will it be adequate to convey the constructs of interest to fourth graders. 

The teacher item could be improved by repeating the context of the question: 

Ask students to do a reading group activity or a project about what they have read? 
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Student Item: 

6. When you have reading assignments in school, how often does your teacher do each of the 
following? Fill in one box on each line. 

Ask you to talk to other students about what you have read? 

Teacher Item: 

How often do you do the following things as a part of reading instruction with this class? Fill in 
one box on each line. 



Ask students to talk with each other about what they have read? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



79% (11 out of 14 responses; 2 missing) 

7 

4 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 


2 


2 


1 


1 


1-2 times/ month 


3 


1 






1-2 times/ week 


1 






3 


Almost every day 
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Discrepancy rate 



29% (2 out of 7 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 2 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 




1 




1-2 times/ week 


1 


3 




Almost every day 


1 




1 



Discussion 

At least five students had trouble estimating the frequency of this activity, which led to both 
under- and overreporting. There were issues with the 4-point response option scale, with the unanchored 
time frame (one student gave an example from 2 nd grade), and with comprehension of the construct of 
interest (i.e., talking to other students about what you had read.) 

Students had various idiosyncratic interpretations of the question: 

♦ Talk in front of the class 

♦ Talking to her friends about what she has read without the teacher asking her to do that. She 
said: "We read every day in school and they are always talking about what they have read and 
I tell them what I have read." 

♦ Asking about book reports 

♦ We never get in pairs — we only do this when we have book reports. 

An ESL student did not understand the question and left it blank. 

Recommendations 

Similar to the previous item, fourth graders do not appear to have a common understanding of the 
construct being assessed. Simplifying the scale seemed to improve some of the frequency estimation 
problems, but this revision might not be sufficient to justify administration of the item to fourth graders. 



The Measurement of Instructional Background Indicators 



127 



Fourth-Grade Reading Students and Teachers 



Student Item: 

7. When you have reading assignments in school, how often does your teacher do each of the 
following? Fill in one box on each line. 

Ask you to write something about what you have read? 

Teacher Item: 

How often do you do the following things as a part of reading instruction with this class? Fill in 
one box on each line. 



Ask students to write about something they have read? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



33% (5 out of 15 responses) 

1 

4 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 










1-2 times/ week 


1 


2 


8 


1 


Almost every day 


1 






2 
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Discrepancy rate 



71% (5 out of 7 responses) 



Number children reporting more frequently 1 

Number children reporting less frequently 4 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 








1-2 times/ week 






1 


Almost every day 




4 


2 



Discussion 

Three students lost the context of the question. Two of these students included social studies and 
science in their responses, respectively, and the third student forgot that the question was asking about 
how often the teacher asks the them to write about something they have read. 

Two students only considered monthly book reports in their answers, which led to 
underreporting. 

One teacher included reading as part of history instruction. However, two of her three students 
provided the same response (“once or twice a week”) by chance. The third student thought “sometimes” 
meant “not that often” and since they do it every week, he checked “almost every day.” 

One student was not able to categorize his judgment into one of the initial four categories. And 
an ESL student misunderstood the question. 



Recommendations 

In spite of the relatively low discrepancy rate, fourth graders do not seem to have a shared 
understanding of the item’s intent. They either define it narrowly (only book reports) or lose context and 
include all types of reading. The issue of loss of context is nearly impossible to resolve, since students 
have reading assignments in social studies, science, and other areas - as well as the intended area of 
reading (only). 

The teacher item could be improved by repeating the context of the question: 

Ask students to write about something they have read during reading instruction? 
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Student Item: 

8. When you have reading assignments in school, how often does your teacher do each of the 
following? Fill in one box on each line. 

Help you break words into parts? 

Teacher Validation Probe (not survey item): 



Help students break words into parts? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



62% (8 out of 13 responses; 3 missing) 

4 

4 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 


2 




2 




1-2 times/ month 










1-2 times/ week 


3 




3 


2 


Almost every day 






1 
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Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



50% (2 out of 4 responses; 3 missing) 
0 
2 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 








1-2 times/ week 


1 


2 




Almost every day 




1 





Discussion 

Most of the discrepancies (at least six) were due to individual variation. That is, some students 
need more help breaking words into parts and others need less. Nonetheless, several problems with the 
item were identified. 

Four students lost the context of the question. For example, one student thought of students 
helping each other break words into parts, as opposed to the teacher. She said: “We never do that - we 
never help each other break the words.” 

Another four students did not understand what “break words into parts” meant. One student 
defined it as “like making sentences, like writing.” Another student chose “never or hardly ever” because 
he did not understand the question. 

At least two students had trouble estimating behavioral frequencies. 



Recommendations 

This item should be deleted. Many fourth graders are unfamiliar with the construct being 
assessed. In addition, this item is an indicator of student reading ability: good readers should report this 
less frequently than poor readers. Accordingly, it cannot be used as a student-level indicator of the 
impact of instructional practices on behaviors. 
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Student Item : 

9. When you have reading assignments in school, how often does your teacher do each of the 
following? Fill in one box on each line. 

Help you understand new words? 

Teacher Item: 

How often do you do the following things as a part of reading instruction with this class? Fill in 
one box on each line. 



Help students understand new words? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



60% (9 out of 15 responses; 1 missing) 

4 

5 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 


1 


1 


1 


1 


1-2 times/ week 




1 


3 


2 


Almost every day 


2 




1 


2 
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Discrepancy rate 



86% (6 out of 7 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 6 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


Almost every 
day 


Never or hardly ever 








1-2 times/ month 








1-2 times/ week 








Almost every day 




6 


1 



Discussion 

Similar to the previous question, most of the discrepancies (about 12) were due to individual 
variation. As an above average student explained: "You stop learning new words in 3rd grade. Some 
kids might learn new words, but not me!” 

Three students lost the context of the question. For instance, one student only responded with 
respect to social studies. 

Four students thought this question was similar to the previous one (“Help you break words into 
parts?”) and answered it the same way. 



Recommendations 

Given the problems students encountered with the previous item, as well as the perceived 
similarity between this item and the previous one, we propose asking fourth graders only this question. In 
order to deal with the loss of context, however, we propose revising it the following way: 

Help you understand new words when you are reading stories? 
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Student Item: 

10. Do you and your teacher review your progress in reading by looking at your work together? 

Teacher Validation Probe (not a survey item): 

Do you review their progress in reading by looking at their work with them? 



Both: 



Yes □ 
No □ 



Discrepancy rate 



14% (3 out of 21 responses; 2 missing) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 


1 


1 


Yes 


2 


17 



Discussion 

In spite of the low discrepancy rate in this item, students had many comprehension problems. 
What specifically seemed to throw them off was the word “progress.” In fact, one student skipped the 
question because he did not understand “progress.” Two of the students who answered differently than 
their teacher (they both said “no”; their teacher “yes”) paraphrased the question as: 

♦ To know if you have done something with your teacher in progress reading. 

♦ If she were to grade his paper in front of him, with him watching, this would be an example 
of “reviewing progress in reading by looking at your work together.” But the teacher does 
not do this. Instead, the teacher looks over his stuff and returns it with grades or comments. 

Other students, whose responses were in agreement with their teachers’, had various idiosyncratic 
interpretations of the item: 

♦ One student appeared to focus on the last half of the question “looking at your work 
together,” saying that students correct each other's tests. She interpreted “review your 
progress” as “going over and looking up words silently.” “The teacher corrects us in math; in 
reading, we correct papers and she [the teacher] puts on a sticker.” 

♦ Another student defined it as: “Like, review it. Like when someone reads too fast and the 
teacher says ‘say it again’.” The student appeared to have interpreted “review” to mean 
“repeat.” 

♦ Two students interpreted the item to mean “formal review,” specifically parent-teacher 
conferences. As one of the students put it: “We only do this on report card days and open 
house.” 
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Finally, there were too many clauses in the question for one student. She didn’t keep “in reading” 
in her mind when searching for representations of the construct in her memory and included math in her 
response — going over math homework with the teacher. 



Recommendations 

The item does not appear to have much discriminating power. That is, most of the respondents 
answer “yes” to the item. It seems to be common teacher practice to review students’ progress in reading. 
However, some students do not appear to have a clear understanding of the construct in spite of the 
apparent agreement with their teachers’ responses. Consider: 

Does your teacher ever spend a few minutes talking just with you about: 

a) your reading homework , reading tests, or other reading assignments? 

Yes □ 

No □ 
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Student Item: 

1 1 . How often do you use the school library or the public library to do the following? Fill in one 
box on the line. 



Do research for a school assignment? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Teacher Validation Probe (not a survey item): 



Do your students ever use the school library or public library - or do they use the Internet - 
to do research for a school assignment? 

Yes □ 

No □ 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



67% (10 out of 15 responses; 1 missing) 

7 

3 



Teacher 

Responses 


Student Responses 


Never or 
hardly ever 


Yes 


1-2 times/ 
month 


1-2 times/ 
week 


Almost 
every day 


No 


3 


5 


2 




Yes 


2 


3 







O 
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Discrepancy rate 



29% (2 out of 7 responses) 



Number children reporting more frequently 1 

Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


Yes 


Sometimes 


Almost every day 


No 


1 


1 




Yes 


1 


4 





Discussion 

Similar to earlier items, some of the discrepancies (a total of four) were due to individual 
variation. The item had a few problems, however. 

Two students were more inclusive in their responses than their teachers. One student included 
checking out books for book reports and doing homework at the library. The other student counted any 
time spent in the library. A third student included electronic search in her response. Her teacher 
indicated that the students do not use the library for research, but they do use the Internet. So, in reality 
the student and teacher were in agreement. 

One student misinterpreted the question, which led to disagreement with her teacher. She defined 
“do research for a school assignment” as: “Like get a book and read it and see if that’s the one you want.” 
At least one other student, in spite of response agreement with the teacher, did not know what “research” 
or “assignment” meant. 



Recommendations 

Given that some students do not seem to have a common understanding of what is meant by “do 
research for a school assignment,” the construct needs to be defined more clearly. We are unsure of the 
true intent of the item. 
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Student Item: 

12. This year in school, how often have you been asked to write long answers to questions on 
tests or assignments that involved reading? 

Teacher Validation Probes (not survey items): 

Thinking of the last reading test or assignment -- tell me about it. 

What kinds of questions did you ask? Did you ask questions that required long written 
responses? Are these pretty typical of your reading tests and assignments? About what 
proportion of your reading tests and assignments have questions that require students to 
write long answers? About how often do you give reading tests and assignments? 



Original scale: 




Revised scale: 




Almost every day 


□ 


Almost every day 


□ 


Once or twice a week 


□ 


Sometimes 


□ 


Once or twice a month 


□ 


Never 


□ 


Never or hardly ever 


□ 







Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



67% (8 out of 12 responses; 5 missing) 

2 

6 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ 
year 


1-2 times/ 
month 


At least once a week 


Never or hardly 
ever 






1 


1 


1-2 times/ year 










1-2 times/ month 










At least once a 
week 


i 




5 


4 
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Discrepancy rate 



83% (5 out of 6 responses) 



Number children reporting more frequently 4 

Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never 


Sometimes 


At least once a 
week 


Never or hardly ever 




i 


2 


1-2 times/ year 








1 -2 times/ month 




i 


1 


At least once a week 




i 





Discussion 

One teacher was not asked the validation probes, which is the reason that five student responses 
are missing. Of these five students, two students left the question blank - one because he ran out of time 
and the other because she did not understand the question. 

At least seven students did not have a clear understanding of the question. Two of these students 
defined “long answers” as “long sentences.” Another student paraphrased the question as: “How many 
times have you done reading and spelling tests?” 

Five students lost the context of the question. They responded with respect to other subjects 
(math, social studies, and science) or other grades (3 rd grade). It can be argued that the term “tests or 
assignments that involved reading” does not only apply to reading but can also include other subjects, 
such as reading in social studies. Thus, these students’ interpretations are textually supported. 

The revised scale (at least once a week; sometimes; never) does not appear to be an improvement 
over the original scale, because the response options “at least once a week” and “sometimes” are not 
mutually exclusive. In fact, the original scale (at least once a week; once or twice a month ; once or twice 
a year; never or hardly ever) might be easier for students to answer because it has less discriminating 
power at the most frequent end compared to the other four-point scale ( almost every day; once or twice a 
week; once or twice a month; never or hardly ever). 



Recommendations 

Fourth graders do not have a common understanding of what is meant by “write long answers to 
questions on tests or assignments that involved reading.” In addition, the question is not specifically 
tailored to reading but can be answered with respect to social studies or other subjects, as well. Consider: 

How often do you have to write paragraphs for English or Reading? 
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Teacher Item Only: 

1. How often do you do the following things as a part of reading instruction with this class? Fill 
in one box on each line. 

Ask students to make predictions about what they read as they are reading it? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Almost every day 


2 


Once or twice a week 


2 


Once or twice a month 


2 


Never or hardly ever 





Discussion 

Two of the six teachers chose a more frequent option than their validated response due to social 
desirability. That is, this behavior (asking students to make predictions about what they are reading) was 
seen as a socially desirable teaching technique by these teachers. When adjusting their response during 
the validation part of the interview, they both ended up selecting the most frequent of the two options they 
were choosing between. 

One teacher had difficulty verbalizing the difference between this item and the item that follows 
(“Ask students to make generalizations and draw inferences based on what they have read”). In thinking 
out loud and discussing what she thought this item was asking, she used the term “inferences.” However, 
when asked about the difference between these items, she decided that the differences referred to the 
“level of knowledge of the guess” and then provided different definitions. 



Recommendations 

This item might be prone to social desirability and should be administered and analyzed with 

caution. 
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Teacher Item Only: 

2. How often do you do the following things as a part of reading instruction with this class? Fill 
in one box on each line. 

Ask students to make generalizations and draw inferences based on what they have read? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Almost every day 


2 


Once or twice a week 


2 


Once or twice a month 


2 


Never or hardly ever 





Discussion 

One teacher lost context and included social studies reading in her response. 

Recommendations 

Consider revising the item to avoid the loss of context: 

Ask students to make generalizations and draw inferences based on what they have read 
during reading lessons? 
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Discussion of Fourth Grade Reading Findings 

Discrepancy rates. Discrepancy rates for selected items are presented in table 7 (below). These 
items were selected because they tended to reflect classroom practices rather than individual differences. 
That is, all students in a class were expected to engage in these behaviors with comparable frequencies. 
For these selected items, the use of a simpler rating scale, using vague quantifiers, was not generally 
associated with lower discrepancy rates. Regardless of the scale, discrepancy rates were high. Reasons 
for these high discrepancy rates are very similar to reasons for high discrepancy rates for fourth grade 
mathematics and science items: 

♦ many of the behaviors of concern are not interpreted as intended by the item writers, and 

♦ many fourth graders lack the cognitive abilities to accurately synthesize a behavioral 
frequency, particularly the time period (i.e., the denominator for rate calculations) is either 
ambiguous, unspecified, or long. 

The reasons for these (and other discrepancies) are discussed below. 



Table 7. Discrepancy Rate for Fourth Grade Reading Items, by Type of Scale 



When you have reading assignments in school, how often does your 
teacher do each of the following: 


4-point 

scale 

(n=16) 


3-point 
scale (n=7) 


Ask you to do a group activity or project about what you have read 


50% 


57% 


Ask you to talk to other students about what you have read 


79% 


29% 


As you to write about something you have read 


33% 


71% 


Help you break words into parts 


62% 


50% 


Help you understand new words 


60% 


86% 








This year in school, how often have you been asked to write long 
answers to questions on tests or assignments that involved reading? 


67% 


83% 



NOTE: The number of students (n) is the modal number of respondents to each item. 



Nature of reading instruction. Reading practice and instruction is not restricted to formal 
reading lessons. It occurs for all subjects, and is particularly frequent in history, science, and social 
studies. (When asked about reading instruction, one teacher asked if she should include “Spanish”!) 

Fourth graders would include social studies and science incidents in their responses to the 
following items: 

When you have reading assignments in school, how often does your teacher do each of the 

following? 

Help you understand new words? 

Ask you to write something about what you have read? 

It is not clear if this represented a loss of context or a more inclusive definition of reading assignments 
than was intended by the item writer. 

Time frame issues. With unanchored time frames — that is, items which ask, “When you have 
reading assignments in school, how often does your teacher do each of the following?,” there is no 
explicit demand to restrict reporting to the current grade. Although most students responded about their 
current teacher, some students would base their rate estimates on reading assignments in other grades. 
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Comprehension. Many of the words and phrases used in these items were not familiar to all 
fourth graders. Words and phrases that were inconsistently interpreted by fourth grade students included: 

♦ read on your own 

♦ read for fun 

♦ novel 

♦ group activity or project 

♦ (teacher) ask you to talk to other students about what you have read 

♦ break words into parts 

♦ review your progress in reading 

♦ do research for a school assignment 

♦ write long answers to questions on tests or assignments that involved reading 

The phrase “read on your own” could be interpreted as either referring to reading without anyone else’s 
assistance or as reading on your own volition. “During the past month,” as in the question, “During the 
past month, how many books have you read on your own outside of school?” was responded to with 
reports of behavior in the past, in general; the current month; or the intended construct (the past 30 days). 
Several students retrieved events in the past week and reported only about books read in the past week. 

Complex sentences were very difficult for fourth grade students to understand. For the following 
item, fourth graders failed to internalize all of the aspects of the construct being asked about and were 
unable to retrieve from memory all of the desired types of relevant representations: 



This year in school, how often have you been asked to write long answers to questions on 
tests or assignments that involved reading? 

The students’ agreement with their teachers’ responses was at the levels expected by chance. 

Loss of context. When items are presented in list format, students and teachers often lose 
context. They respond to the listed option as a stand-alone item instead of responding with respect to the 
conditions specified in the stem. For example, teachers would report about reading instruction in the 
context of other subjects when they were asked: 

How often do you do the following things as a part of reading instruction with this class? 

Ask students to do a group activity or project about what they have read? 

Ask students to make generalizations and draw inferences based on what they have 
read? 
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Eighth Grade Science Students and Teachers 

Student Item Only: 

1. How much do you agree with each of the following statements? Fill in one box on each line. 

I am good at science. 

Agree □ 

Not sure □ 

Disagree □ 



Student Responses 


Frequency 


Agree 


6 


Not sure 


4 


Disagree 


1 



Discussion 

Several of the students went straight to the body of this question and skipped the stem “How 
much do you agree with the following statements?” However, for this question (and the following 
question), ignoring the stem had little impact. 

Students used a variety of criteria for deciding whether they were good at science. These 
included assessment of self-competency (“because the teacher gives me stuff and I understand it”) and 
grade reporting (“because I get good grades in it.”) These are similar but not identical factors. One 
student disagreed with the statement because “my science teacher doesn’t teach us well.” 

Recommendations 

If the item is intended to determine if a student enjoys science, it should ask, “Do you like 
science?” If the item is intended to determine if a student gets good grades in science, it should ask, “Do 
you get good grades in science?” If the item is intended to determine if a student thinks science is easy to 
understand, it should ask: “Is science hard or easy for you to understand?” 
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Student Item Only: 

2. How much do you agree with each of the following statements? Fill in one box on each line. 



Science is useful for solving everyday problems. 

Agree □ 

Not sure □ 

Disagree □ 



Student Responses 


Frequency 


Agree 


8 


Not sure 


1 


Disagree 


2 



Discussion 

One student disagreed because “sometimes we need science, but not every day.” He defined 
“everyday problems” as either problems with a friend or science problems you get every day (i.e., class 
work). Other students expressed similar confusion. Another student agreed because she “sometimes 
wonders about stuff in the wild.” A third student said “not sure” because he “could not think of examples 
of everyday problems that related to science.” 



Recommendations 

The intent of this item is not clear. We do not know whether “everyday problems” refers to 
problems that occur in the normal daily life of an eighth grader, to “real-world problems,” such as curing 
diseases, or to something else. Unless the intent can be clarified, this item should be deleted. 
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Student Item: 

3. Have you ever done hands-on activities or projects in school with any of the following? Fill 
in all boxes that apply. 

Living things (for example, plants, animals, bacteria) 

Teacher Validation Probe (not a survey item): 

Your students will be asked about certain kinds of hands-on activities and projects they may have 
done in science. 

Have they done any such activities with living things? 



Discrepancy rate 



91% (10 out of 11 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 


|| | 


10 


Yes 




3 § ; 



Discussion 

An unspecified timeframe and the lack of restrictions on where these activities occurred were the 
main reasons for the ten discrepancies between the students’ and teachers’ answers. Students who 
checked “yes” were thinking either of a different grade or a different class. These students were using an 
appropriate interpretation of the phrase “Have you ever done...” and including activities with living things 
in sixth or seventh grade and activities with living things in eighth grade lab and ecology class. 

Most students did seem to understand the term “living things.” However, one student described 
seeing organs of animals (who used to be living) as an example of living things. The second student 
accurately recalled the seed germination activity in her 8 th grade science class, but the teacher evidently 
forgot this activity. 

One student did not understand the question and left it blank. This is a general problem with 
question formats that ask students to check “all that apply.” If a box is unchecked, it is impossible to 
know whether the student did not do that activity or did not understand the question. Thus, the resulting 
data can be difficult to interpret. 

Recommendations 

Respondents can generally provide more accurate behavioral frequency data for a recent, short, 
well-defined period (i.e., “In your science class this year”) rather than a more ambiguous, all 
encompassing context and period of time (“Have you ever...”). If the intent of the item is to determine 
whether students have ever engaged in these behaviors, the item needs to be reworded to make this point 
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more clear. (Because most of the items dealt with their eighth grade science class, some, but not all, 
students responded in terms of eighth grade science). 

If these items are intended to get at the frequency of behaviors in science class, the wording of the 
stem should be changed to: 

In your science class this year, have you done hands-on activities with any living things (for 

example, plants, animals)? 

To deal with the issue of interpreting whether a blank is a “No,” a “Don’t Know,” or a “Refused 
to answer,” this item should be made into a “Yes/No” item. 
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Student Item: 

4. Have you ever done hands-on activities or projects in school with any of the following? Fill 
in all boxes that apply. 

Magnifying glass or microscope (for looking at small things)? 

Teacher Validation Probe (not a survey item): 

Your students will be asked about certain kinds of hands-on activities and projects they may have 
done in science. 

Have they worked with magnifying glasses or microscopes? 



Discrepancy rate 



55% (6 out of 1 1 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 




6 


Yes 




5 



Discussion 

Students seemed to understand the words “magnifying glass” and “microscope” and gave 
appropriate examples of activities using these objects (e.g., looking at sugar, baking soda, bacteria and 
salt under a microscope, lighting a match under a magnifying glass). One teacher did report the use of 
magnifying glasses and microscopes in the classroom (looking at bone marrow under a microscope and 
magnifying glass). 

However, all of the students reported these classroom activities, while only one teacher reported 
the use of magnifying glasses and microscopes as part of their instructional practices. The teachers did not 
comment further, except one teacher said he used magnifying glasses and microscopes last year. As in 
the previous question, the students did not realize that the item was intended to refer only to their eighth 
grade science class. Some of the students were thinking of activities in science lab, in other grades, or in 
other classes. 
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Recommendations 

The meaning of the words “magnifying glass” and “microscope” are understood by eighth grade 
students, but the intended time frame and location restrictions (this year and science class) are unclear 
because they are not explicit. To correct these problems and to eliminate the problem with blank answers, 
the item should be changed to: 

In your science class this year, have you used a magnifying glass or microscope to look at 
small things? 

Yes □ 

No □ 



The Measurement of Instructional Background Indicators 



149 



157 



Eighth Grade Science Students and Teachers 



Student Item: 

5. Have you ever done hands-on activities or projects in school with any of the following? Fill 
in all boxes that apply. 

Thermometer or barometer (for making measurements)? 

Teacher Validation Probe (not a survey item): 

Your students will be asked about certain kinds of hands-on activities and projects they may have 
done in science. 

Have they done anything with thermometers or barometers? 



Discrepancy rate 



36% (4 out of 1 1 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 


3 


2 


Yes 


2 


4 



Discussion 

This question was accurately interpreted by most eighth graders. Most students accurately 
remembered hands-on activities with thermometers, or the lack of such activities in their eighth grade 
science class. 

The discrepancies between teachers’ and students’ responses were due to a student not 
understanding the question and thus leaving it blank; to a student being certain that his eighth grade 
science class did not work with barometers (but not commenting on work with thermometers); a student 
being sure his eighth grade class did work with thermometers when his teacher said thermometers were 
not used in class; and a student including activities done in sixth grade. Two of the teachers had the 
students use thermometers to measure the temperature of liquids. The use of barometers was not 
mentioned by any of the teachers. 

Most of the students reported that they did not know the meaning of the word “barometer.” One 
child said he thought it had something to do with the weather, but didn’t know what it was. This child did 
not attend to the word “thermometer” and so did not answer the question (see discussion of 
underreporting above). 
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Recommendations 

Most students knew what a thermometer was. Since none of the teachers used barometers in their 
classrooms, it can probably be deleted. The item can be simplified to: 

In your science class this year, have you taken measurements with thermometers? 

Yes □ 

No □ 

This proposed revision also eliminates the analytic ambiguity presented by unanswered items. 
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Student Item: 

6. Have you ever done hands-on activities or projects in school with any of the following? Fill 
in all boxes that apply. 

Simple machines (for example, pulleys and levers)? 

Teacher Validation Probe (not a survey item): 

Your students will be asked about certain kinds of hands-on activities and projects they may have 
done in science. 

Have they done anything with simple machines? 



Discrepancy rate 



55% (6 out of 1 1 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 


5 


6 


Yes 







Discussion 

None of the teachers reported the use of simple machines in their classes. However, six of the 
students reported hands-on activities with simple machines. Two students said “yes,” but did not know 
what was meant by pulleys and levers (for one student, his teacher did show a video on how to make 
small cars do things — which the student could have misconstrued as a hands-on activity with simple 
machines). Two other students said “yes,” but were reporting on activities performed in the fifth grade. 

A fifth student reported on the use of Lego building blocks as hands-on activities with simple machines, 
and a sixth student described an activity where they had to use a pulley to pull up a book. Thus, problems 
with word comprehension and an unspecified time frame contributed to overreporting of students’ 
responses. 

Most of the students did not understand the phrases “simple machines,” and the words “pulley” 
and “lever.” One child thought a lever was “something used to balance things.” 



Recommendations 

It is not clear whether this question can be asked at all. Very few students, if any, understood the 
question, and it would be difficult to simplify the concepts of pulleys and levers. The teachers did not 
report the occurrence of these activities in the classroom. 
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Student Item: 

7. Which best describes the science course you are taking? 

□ I am not taking a science course this year 

□ Life science (for example, biology) 

□ Physical science (for example, physics or chemistry) 

□ Earth science (for example, geology or astronomy) 

□ General science (several content areas of science taught separately) 

□ Integrated science (several content areas of science combined and taught together 
throughout the year) 

Teacher Validation: 

No specific validation probe. Teachers were asked informally about the science course they were 
teaching. They all responded “general science.” 



Discrepancy rate 



67% (6 out of 9 responses; 2 missing) 



Teacher 

Responses 


Student Responses 


1 am not taking a 
science course 
this year 


Life 

science 


Physical 

science 


Earth 

science 


General 

science 


Integrated 

science 


Checked 
two or more 
boxes 


I am not taking a 
science course 
this year 
















Life science 
















Physical science 
















Earth science 
















General science 




1 






3 


1 


4 


Integrated science 
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Discussion 

Two students skipped the item. One of these students did not know what was meant by “course,” 
by “content area,” or by most of the course names (e.g., “physical science” and “integrated science”). 
“Content area” was defined as “content of a book — like a part of a book.” The other student was not sure 
how to answer this question and would therefore skip it. She said: “If I had learned about it, I would 
check it off like a check list.” She didn’t know what “integrated science” meant and thought “general 
science” meant: “without the teacher changing what it’s supposed to be.” She read “physics” as 
“psychics.” 

Four students checked two or more boxes. In fact, one student checked three boxes and another 
student four boxes. The latter student went down the list and chose all of the topics they had covered this 
year (life science, physical science, earth science, and integrated science). He said “We have one class 
that covers all this stuff.” One of the students who checked two boxes chose “life science” and “earth 
science,” since this represented things that he thought he had studied this year. The teacher indicated that 
they had studied the human body and ecology. It is possible that the student interpreted “earth science” to 
be the same thing as ecology. Finally, another student who only checked one box thought it was possible 
to choose more than one. 

At least six students did not know what “integrated science” meant. However, in spite of their 
lack of comprehension, two of these student chose that option and defined it as: 

♦ We haven’t done one specific course this year, we have done a few such as the circulatory and 
reproductive system, sex education, and the periodic table. 

♦ Integrated science is low level; general science is a step higher. 

Another student defined “integrated science as “the mobility of joints.” 

One student had trouble understanding the term “general science.” He defined it as “something 
you learn every year (in 6th, 7th, and 8th grade), like formulas or sex education.” 



Recommendations 

Asking eighth graders to describe their science class in terms of the listed categories does not 
work. The category labels do not correspond to the retrieved representations of the type of class that 
students encode. If the question is administered to eighth grade students, it should be revised: it should be 
specified that only one option can be checked and the response options should be simplified. 
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Student Item: 

8. About how often do you study science in school? 

□ Every day 

□ 3 or 4 times a week 

□ 1 or 2 times a week 

□ Less than once a week 

□ Never 



Teacher Validation: 

No specific validation probe. Teachers were asked informally about how often they taught their 
current science course. 



Discrepancy rate 



36% (4 out of 1 1 responses) 



Number children reporting more frequently 3 
Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never 


Less than 
once a week 


1 or 2 times a 
week 


3 or 4 times a 
week 


Every day 


Never 












Less than once a week 












1 or 2 times a week 












3 or 4 times a week 






1 


4 


3 


Every day 










3 



Discussion 

The three students who overreported how often they have science included “science lab” in their 
responses, whereas their teacher did not. The class has science lab one day per week. If studying science 
in school includes science labs, these students’ responses are correct. However, the responses of their 
peers who did not include science lab would then be incorrect. 

Two students understood the item as: “How often do you study?” That is, they interpreted the 
item to ask about how often they study science, like how many times they research or review things in 
science, as opposed to how many times their science class meets. Even though one of the student’s 
response matched the teacher’s response, this agreement was coincidental. 
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Recommendations 

If the intent of the item is to determine how often the student’s science class meets, it needs to be 
reworded to ask this. If not, it needs to be reworded so that other students will include science lab 
courses. 

The word “study” is confusing to some students. The item could be reworded to: “ How often do 
you have science in school ?” or “How often does your science class meet ?” However, the intent of the 
item (see above) needs to be ascertained before revision is attempted. 
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Student Item: 

9. Do either you or your teacher save your science work in a portfolio? 

Teacher Validation Probe (not a survey item): 

Do you or your students keep portfolios of their work? 



Both: 



Yes □ 
No □ 



Discrepancy rate 



55% (6 out of 1 1 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 


2 


6 


Yes 




3 



Discussion 

The term “portfolio” has very different meanings to teachers and students. The students’ 
definitions were usually more inclusive than their teachers. Students gave various definitions of 
portfolio: 

♦ She (the student) told about her teacher saving some work on a project, in process, in a 
portfolio. The student was talking about keeping this work over night, so that it could be 
handed back to the students for completion the next day. 

♦ He (the student) was thinking of one project that involved the teacher collecting all of the 
students’ work, then scanning them into a portfolio. 

♦ She (the student) defined a portfolio as a place where the teacher keeps the students’ work. 
The teacher gave the students their work back at the last day of school. 

♦ Teachers defined a portfolio more narrowly as a set of documents representing the students’ 
eighth grade science work in a folder. All but two students felt the teacher did keep their 
work in a portfolio, as they more generally defined the term. 

One teacher mentioned that the school has a portfolio for all four eighth grade subjects, but not 
one for science only. 
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Recommendations 

Clearly, the term “portfolio” needs to be more explicit. If the teacher defines it as a specific 
collection of the work done by the student, the question will need to be more narrowly constructed around 
that definition. A complete definition, with examples, should be given to more clearly define the term. In 
order to provide such a definition, further guidance about the type of portfolio (and its use or uses) to be 
measured is necessary. 
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Student Item: 

10. If you are taking a science course this year, about how much time do you spend doing science 
homework each week? 



I am not taking a science course □ 
None □ 

Vi hour □ 

1 hour □ 

2 hours □ 

3 hours □ 

More than 3 hours □ 



Teacher Validation Probes (not survey items): 

We’re also going to ask them about science homework. 

How often do you assign science homework? 

days per week 

About how long do you think it takes your students to do their science homework? 
minutes per assignment 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



82% (9 out of 1 1 responses) 
0 
9 
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Teacher 

Responses 

(per week) 


Student Responses (per week) 


1 am not 
taking 
science 
this year 


None 


V 2 hour 


1 hour 


2 hours 


3 hours 


More than 
3 hours 


I am not taking 
science this 
year 
















None 
















Vi hour 




2 


1 










1 hour 
















2 hours 






3 


4 


1 






3 hours 
















More than 3 
hours 

















NOTE: Teacher responses to the above probes were combined into a single teacher response. 



Discussion 

There are several sources of possible discrepancies between students’ and teachers’ responses. 
The amount of homework teachers assign varies greatly over the course of the year. Indeed, teachers 
reported difficulty in estimating the amount of homework because the levels vary so much within the 
science classes. In addition, students’ abilities in science will cause individual variation in the time it 
takes to complete the same homework assignment. Students’ answers sometimes varied from that of the 
teachers because the students simply completed the homework more quickly than the teacher anticipated. 

In addition, the time period of the responses, ranging from “V4 hour per week” to “more than 3 
hours per week,” may not be optimal for science classes where homework can vary greatly from week to 
week. 



Seven students underreported the amount of time it takes them to complete their homework as 
compared to the teachers because their teachers assigned homework only every month or so. Two of 
these teachers’ students chose the option, “none.” The remaining seven students who reported less 
homework than their teachers (who reported 2 hours per week of homework) had varying problems with 
or misinterpretations of the item: 

♦ reported V 2 hour of homework, but didn’t know whether or not to include project work 

♦ reported V 2 hour of homework, but was an “A” student, so probably completed his work more 
quickly 

♦ reported Vi hour of homework, but indicated that they usually don’t get difficult assignments 

♦ two students reported 1 hour of homework, but didn’t know whether or not to include project 
work 

♦ reported 1 hour of homework, but admitted to individual variation among students 
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♦ reported 1 hour of homework, but was thinking of how much homework he anticipates for 
next year (he had been reading a brochure on the curriculum of the high school he was 
attending) 

No students overreported the amount of homework assigned. It is possible that teachers were 
“generous” with their estimates — or that they averaged projects into their estimates. 

Confusions also resulted from various interpretation of “homework.” Some projects extend after 
school hours and so were interpreted by some as homework. 



Recommendations 

Science homework appears to be comprised of at least two distinct components: “Regular” 
homework and “Project homework.” Further, science homework appears to be episodic, making it 
difficult for respondents to provide estimates for the average week. 

This item would need to be quite specific in order to be more accurate. “Homework” should be 
carefully defined and examples given. Sacrifices in estimates over the entire year may have to be made in 
order to elicit more accurate answers. For example, the students may need to be asked: 

How much time did you spend last week on science homework? Do not count project work . 



I am not taking science this year □ 

I did not do any science homework last week □ 
About V 2 hour □ 

About 1 hour □ 

About 2 hours □ 

About 3 hours □ 

More than 3 hours □ 



How much time outside of the classroom did you spend last week on science projects? 



I am not taking science this year □ 

I did not work on any science projects last week □ 

About % hour □ 

About 1 hour □ 

About 2 hours □ 

About 3 hours □ 

More than 3 hours □ 
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Student Item: 

1 1. Do you ever do science projects in school that take a week or more? 

Teacher Validation Probe (not a survey item): 

Do your students ever do science projects in school that take a week or more? 



Both: 



Yes □ 
No □ 



Discrepancy rate 



27% (3 out of 1 1 responses) 



Teacher 

Responses 


Student 

responses 


No 


Yes 


No 




3 


Yes 




8 



Discussion 

In the participating schools, long projects are quite common in eighth grade science class. Even 
the one teacher who ended up saying that he did not give out long projects, had a difficult time making 
the decision, and, in fact, described a long-term project where students were given sacks of flour to care 
for as they would newborn babies. 



Recommendations 

More explicitly defining the phrase “science projects” might be helpful to increase accuracy of 
the responses. Conversely, if everyone is answering “yes,” the item may not have any discriminating 
power and can be deleted. 



0 
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Student Item: 

12. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Teacher Item: 

About how often do your science students do each of the following? Fill in one box on each line. 



Both: 



Discuss science in the news? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 18% (2 out of 1 1 responses) 

Number children reporting more frequently 1 

Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 


6 








1-2 times/ month 


1 


3 


1 




1-2 times/ week 










Almost every day 











Discussion 

According to most of the teachers and students, this activity was not very common. However, 
definitions of “science in the news” varied, with teachers generally having a more inclusive definition of 
this activity. One teacher described it as “any information from the news that is relevant for a student’s 
science fair project;” another, as “science one hears about through the mass media (cloning, ecological 
issues)” whether or not it was related to current student projects. Students defined “in the news” as 
“having to do with media,” “science shows seen on TV,” discussing any news item in science class (even 
if it is not science-related), and listening to the news about science. One student underreported since she 
had a more narrow definition of “science in the news” than her teacher. Another student overreported this 
activity because he was including science TV programs he watches outside of class. However, since none 
of these activities occurred very often, the teachers’ and students’ responses generally were consistent. 
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One general problem is that students find it difficult to retain the proper context of the answers. 
Students forget to read the stem “When you study science in school” and often answer questions by 
including events outside of the classroom, or outside of the relevant subject area. One student skipped the 
directions, was very confused, then went back and read the directions, changing her answer. 

Finally, many students interpreted “Never or hardly ever” to mean “Never.” When responding 
about infrequent events, students often pick the nearest response to “Never or hardly ever.” Thus, 
students may answer “once or twice a month,” even when the event occurs only every few months. 



Recommendations 

Providing an example to illustrate what is meant by “discuss science in the news” may improve 
the validity of reporting. However, students often respond only in terms of the examples; the examples 
focus the responses too narrowly. Reinstating the context (of the event occurring when science is studied 
in school) may also improve the validity of the responses: 



In your science class this year, how often do you discuss events in the news that are related to 
what you are learning in science class? 



Every science class □ 

Once or twice a week □ 

Once or twice a month □ 

Hardly ever □ 

Never □ 



( These two scale choices allow students to accurately 
report infrequent events .) 
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Student Item: 

13. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Teacher Item: 

About how often do your science students do each of the following? Fill in one box on each line. 



Both: 



Work with other students on a science activity or project? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



82% (9 out of 1 1 responses) 



Number children reporting more frequently 2 
Number children reporting less frequently 7 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 










1-2 times/ week 




1 




2 


Almost every day 




3 


3 


2 



Discussion 

This question is a good example of a general problem with rate estimates produced by teachers 
and students. It is difficult for students to estimate the average frequency for an activity. These 
estimations are especially difficult when the time period is undefined. Should the frequency be averaged 
over the last week, the last few months, the school year or only for the period during which the activity 
was being performed? Many students, even teachers, are influenced greatly by recent events, and thus do 
not accurately estimate over long periods of time. One teacher responded that, lately, his students work 
with each other on a science activity “one to two times per week.” However, he added that the yearly 
average would be “almost every day.” 

Estimating the frequencies of events is especially difficult for items asking about subject areas 
that are not taught daily. The response option “Almost every day” could be interpreted as “almost every 
day we have science (we do that activity)” or “almost every day of the week (we do that activity).” Much 
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of the underreporting of frequencies by students was due to the fact that the students do not have science 
every day. Some students used a literal interpretation of “Almost every day” and thus did not use that 
option. However, their teacher interpreted “Almost every day” to mean “almost every day that science 
class is taught.” 

Underreporting was also due to students thinking only of projects that are not assigned very 
frequently. One teacher responded “almost every day,” saying that every student has to teach a lesson 
during each lab activity. Finally, one student did not include science lab activities, whereas her teacher 
did. 

Recommendations 

The following version of this item specifies the time period for estimation (although students’ 
responses may still be influenced by the recency of activities), and specifies the meaning of “every day” 
for classes that are not taught daily. Examples may help to explain what is meant by activities and 
projects, but still has the problem of focusing the students only on the listed projects or activities. 

In your science class this year, how often do you work with other students on a science activity 
or project? 

Almost every science class □ 

Once or twice a week □ 

Once or twice a month □ 

Hardly ever □ 

Never □ 
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Student Item: 

14. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Teacher Item: 

About how often do your science students do each of the following? Fill in one box on each line. 



Both: 



Give an oral science report? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



55% (6 out of 1 1 responses) 



Number children reporting more frequently 6 
Number children reporting less frequently 0 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


5 


4 


1 


1 


1-2 times/ month 










1-2 times/ week 










Almost every day 











Discussion 

This item points to several interesting problems resulting in discrepancies between students’ and 
teachers’ responses. For infrequent events, students and teachers alike found it difficult to fit their 
answers into the available response options. One student said the class gave oral science reports at the 
beginning of the year, but not now. He had difficulty choosing an appropriate response category for this 
frequency. Another student reported that his class gave oral reports “maybe once or twice the entire 
year,” and another said “once per quarter.” All these students had difficulty in finding an appropriate 
category that fit the frequency of the event. Four chose “once or twice a month,” whereas their teachers 
chose “never or hardly ever.” One teacher even explained that his class gave oral reports three times a 
year for science fair projects, along with oral reports and explaining things to class. This teacher chose 
“never or hardly ever.” One of his students reported these activities took place “once or twice a month.” 
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Another problem that resulted in discrepancies between teachers’ and students’ responses was the 
interpretation of “giving an oral report.” Some students defined this activity informally (talking to other 
students, writing a letter about cells to a teacher, talking about chapters), while teachers defined the 
activity as covering only formal presentations (students standing in front of the class and giving a report). 
This led to overreporting by students who defined the activity more informally. 

Recommendations 

The response options should explicitly allow for activities that occur less frequently than every 
month. If a more formal definition (of giving an oral science report) is agreed upon, then the question 
might be rephrased as follows. 

In your science class this year, how often do you stand up in front of the class and give an oral 
science report? 



Almost every science class □ 

Once or twice a week □ 

Once or twice a month □ 

Only a few times a year □ 

Hardly ever □ 

Never □ 
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Student Item: 

15. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Teacher Item: 

About how often do your science students do each of the following? Fill in one box on each line. 

Both: 

Do hands-on activities or investigations in science? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



73% (8 out of 1 1 responses) 



Number children reporting more frequently 0 
Number children reporting less frequently 8 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 










1-2 times/ week 


3 


5 


3 




Almost every day 











Discussion 

For many students, this item was difficult to comprehend. Students did not understand the phrase 
“investigations in science.” Some defined it vaguely as: “finding out facts,” “looking for information 
about something in science,” “doing research,” “doing research outside of class,” “researching how they 
do science,” and “reading a book.” Other students simply had no idea of what the phrase meant. For 
these reasons, students underreported the frequency of what they thought were “investigations in 
science.” 



Students also underreported because they ignored one part of the two part question, focusing 
either on “investigations in science” or “hands-on activities,” but not both. 
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Finally, teachers and students alike had difficulty choosing between the response option “once or 
twice a month” and “once or twice a week.” Often, their estimates fell between these points. One teacher 
expressed that he was “waffling” between the two response options. 

One teacher also expressed that the frequency of these activities depends on the unit (i.e., 
chemistry, astronomy, physics, ecology). To answer about science “in general,” with respect to this 
activity, was difficult. 

Recommendations 

The term “investigations” needs to be defined precisely, and then simplified -- or eliminated. 

One possibility for simplifying the question, is: 

About how often do you do hands-on activities in your science class? 

Almost every science class □ 

Once or twice a week □ 

Once or twice a month □ 

Only a few times a year □ 

Hardly ever □ 

Never □ 
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Student Item: 

16. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Talk about the measurements and results from your hands-on activities or 
investigations? 

Teacher Item: 

About how often do your science students do each of the following? Fill in one box on each line. 

Talk about measurements and results from students* hands-on activities? 



Both: 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



82% (9 out of 1 1 responses) 



Number children reporting more frequently 2 
Number children reporting less frequently 7 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 




1 


2 




1-2 times/ week 


3 


4 


1 




Almost every day 











Discussion 

This question is long, and was quite difficult for students to understand. At least one student had 
no idea what it meant. Two students rephrased the question as: “sometimes speakers come and we have 
to measure tables,” and “talking about what you do to investigate things.” 

In addition, this question had some of the same problems as the previous one. Teachers and 
students varied in their interpretations of what was meant by “talk about.” Respondents interpreted the 
phrase to mean either formally talking in front of class or informally after every activity. For this item, 
students often defined the activity more formally, thus resulting in underreporting as compared to their 
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teachers. Some students also focused on either “hands-on activities” or “measurements” but not both 
together. 

One student simply did not understand the question, and chose “never or hardly ever” as her 
answer. When there is no option for “don’t know,” it is difficult to tell if the frequency of the event is 
“never or hardly ever” or if the student does not know the frequency of the event, or does not understand 
the question and has no other option. 

Finally, one teacher felt that “measurements” and “results” were redundant, since “results include 
measurements.” 



Recommendations 

Repeating the question in each response category may help students to answer more difficult 
questions. 

In your science class this year, how often do you talk about results from your science projects 
and other hands-on activities? Include conversations with other students and with your 
teacher. 



I talk about results in almost every science class □ 

I talk about results once or twice a week □ 

I talk about results once or twice a month □ 

I talk about results once or twice a year □ 

I hardly ever talk about results □ 

I never talk about results □ 
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Student Item: 

17. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Teacher Item: 

Do your students... 



Both: 



Go outside to observe or measure things? 

Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



36% (4 out of 1 1 responses) 



Number children reporting more frequently 4 
Number children reporting less frequently 0 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


7 


3 


1 




1-2 times/ month 










1-2 times/ week 










Almost every day 











Discussion 

This item referred to behaviors that were very salient to students. Thus, “going outside to observe 
and measuring things” was easy for students to remember and report. However, the context of the 
question is unanchored. This activity was not part of the eighth grade science classes included in the 
study. Overreporting occurred because students either thought of other classes (i.e., science lab), other 
situations (i.e., a science field trip where they did this once and the student interpreted the “Never or 
hardly ever category” as “Never” and so picked the closest option of “Once or twice a month”), or other 
years. When students were thinking of their science class and the present year, their answers were 
consistent with their teachers. 
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Recommendations 

The question is a good one for students, especially if the context is anchored to a particular time 
and situation. 

For your science class this year, how often do you go outside to observe or measure things? 



Almost every day I have science □ 

Once or twice a week □ 

Once or twice a month □ 

Hardly ever □ 

Never □ 
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Student Item: 

18. When you study science in school, how often do you do each of the following? Fill in one 
box on each line. 

Design and carry out your own science investigation? 

Teacher Item: 

Do your students... 

Design and carry out their own science investigation? 

Both: 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



55% (6 out of 1 1 responses) 



Number children reporting more frequently 3 
Number children reporting less frequently 3 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


5 


2 


1 




1-2 times/ month 










1-2 times/ week 


3 








Almost every day 











Discussion 

Students had difficulty in understanding this question. They had varying interpretations of the 
terms “design and carry out” and “science investigation.” Many students thought that science 
investigation was something that happened only with science projects. Another student included one 
instance of a science project titled “designing a cell” since the project title contained the word “design.” 
Once again, a student interpreted “Never or hardly ever” to mean “Never” and so chose the next, greater 
frequency option (“Once or twice a month”) for an event that occurred much less frequently. Finally, a 
student included activities done in science lab rather than in science class. 
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Teachers’ reporting of this activity in their science classes varied. One teacher reported “once or 
twice per week” (all her students reported “Never or hardly ever”) and the two others reported “Never or 
hardly ever” (many of their students reported once or twice a month or once or twice a week). 

Recommendations 

The construct “design and carry out a scientific investigation” appears to be one that is unfamiliar 
to most eighth graders. It is also interpreted in different ways by different eighth grade teachers. It 
should probably be deleted. 
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Student Item: 

19. When you study science in school, how often does your teacher do each of the following? 
Fill in one box on each line. 

Teacher Item: 

When you teach science in school, about how often do you do each of the following? Fill in one 
box on each line. 



Both: 



Talk to the class about science? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



27% (3 out of 1 1 responses) 



Number children reporting more frequently 0 
Number children reporting less frequently 3 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1 -2 times/ month 










1-2 times/ week 










Almost every day 




1 


2 


8 



Discussion 

The teacher version of this question elicited some very interesting responses: “Of course I talk to 
the class about science. Since I teach science, it’s the focal point of what I talk about - I never just 
lecture,” “I talk to the class about science every time I teach science, I don’t get the point of the question.” 
All teachers responded that they talk to their class about science “Almost every day.” 

Students’ answers, however, varied from “Almost every day” to “Once or twice per month.” One 
student wondered if they were asking about “science or about what they do.” We believe this student was 
trying to articulate the same confusion that the teachers expressed. 



The Measurement of Instructional Background Indicators 



177 



Eighth Grade Science Students and Teachers 



Another student said his science teacher talks to the students only “once or twice per week,” since 
he does not have science class every day. Teachers were responding in terms of the days they taught 
science class. 

Recommendations 

The intent of the teacher question needs to be determined and stated explicitly. If the intent of the 
question is to find out how much time is spend lecturing, as opposed to student-involved activity, then 
perhaps the following sets of questions would be more appropriate: 

Teacher question: 

When you teach science in school about how often do you do each of the following? 

Stand up in front of the class and lecture about science topics? 

Student question: 

In your science class this year ; how often does your teacher lecture to the class about science 
(where you are just sitting and listening)? 



Almost every day I have science □ 

Once or twice a week □ 

Once or twice a month □ 

Hardly ever □ 

Never □ 
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Student Item: 

20. When you study science in school, how often does your teacher do each of the following? 
Fill in one box on each line. 

Teacher Item: 

When you teach science in school, about how often do you do each of the following? Fill in one 
box on each line. 



Both: 



Do a science demonstration? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



45% (5 out of 1 1 responses) 



Number children reporting more frequently 3 
Number children reporting less frequently 2 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


3 




2 




1-2 times/ month 




2 


1 




1-2 times/ week 




2 


1 




Almost every day 











Discussion 

In this question, it was unclear which events the phrase “science demonstration” was intended to 
include. Even the teachers disagreed. One teacher defined it as including demonstrations done either by 
herself or by guest speakers (her students underreported the frequency of this event, because they did not 
include guest speakers). 

Another teacher defined a science demonstration as “showing something in science” and the third 
teacher similarly defined it as “something the teacher does or sets up, where students observe and discuss 
the demonstration.” Two of the students of these teachers overreported because one of them included 
computer demonstrations and the other included science demonstrations done in the science lab. 
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Recommendations 

Examples might help to clarify the intended meaning of the term “science demonstration.” 
However, as mentioned before, examples often focus students on only the specific examples provided. 
Regardless of how the question is worded, a revised scale is recommended: 



Almost every day I have science □ 
Once or twice a week □ 

Once or twice a month □ 

Hardly ever □ 

Never □ 
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Student Item: 

21. When you study science in school, how often does your teacher do each of the following? 
Fill in one box on each line. 

Teacher Item: 

When you teach science in school, about how often do you do each of the following? Fill in one 
box on each line. 



Both: 



Use computers for science (e.g., science software, telecommunications)? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



18% (2 out of 1 1 responses) 



Number children reporting more frequently 1 
Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 


3 








1-2 times/ month 


1 


6 




1 


1-2 times/ week 










Almost every day 











Discussion 

There were several problems associated with this item. Students did not know what “e.g.” meant, 
and the words “telecommunications” and “science software” are not eighth grade vocabulary words. One 
student thought “e.g.” was a computer program! Another thought “telecommunications” was 
“communications with a person about science.” This caused a lot of confusion but resulted in frequent 
coincidental agreement between students and teachers. 

Asking about the use of computers for science in terms of the teachers’ behavior is confusing. It 
was interpreted differentially by teachers as either: 

a) the teacher using the computer or 

b) the teacher allowing their class to use the computer. 
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For teachers who interpreted the item to reflect student behavior, the students’ use of the 
computer for science varied tremendously from student to student. One teacher said that, in general, 
students are using computers in the science class every day, but each student only uses it about once a 
month. (Note: For purposes of validation, the teachers’ computer usage for science is reported in the 
table.) 

Similar to some of the teachers, two students lost the context of the question and responded with 
respect to student computer usage as opposed to teacher usage. 

Recommendations 

Teacher item: 

When you teach science in school, about how often do your students do each of the 
following? Fill in one box on each line . 

Use computers for science (e.g., science software, communicating with others about 
science, researching science topics ). 

(It has to be clarified whether the teacher is supposed to respond with respect to any 
student or with respect to the typical student.) 

Student item: 

In your science class this year, do you use computers? 
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Student Item: 

22. About how often does your science class go on a science field trip? 

Teacher Item: 

About how often do your science students go on a science field trip? 



Both: 



3 or more times a year □ 

1 or 2 times a year □ 

Never or hardly ever □ 



Discrepancy rate 



18% (2 out of 1 1 responses) 



Number children reporting more frequently 1 
Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never or hardly ever 


1-2 times/ year 


3 or more times/ year 


Never or hardly ever 


3 






1-2 times/ year 


1 


6 


1 


3 or more times/ year 









Discussion 

This question can be reliably answered by eighth grade students, perhaps because it occurs 
infrequently, and is a very, salient event. In one case, even though the item was stated in terms of 
“science class” and “science field trips,” a student said he went on field trips “3 or more times per year,” 
including field trips taken with other classes. Another student responded that she never went on Science 
field trips, when her teacher had taken them to Great America for Physics day. It could be that she was 
not present on this trip, forgot about it, or did not consider it science. 
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Recommendations 

The middle and last response options are not mutually exclusive. If a student went on one field 
trip, (s)he could either check that (s)he “hardly ever” went on field trips, or that (s)he went on field trips 
“1 or 2 times a year.” The third option should be simplified to “Never.” 

How often do you go on science field trips with your science class? 

3 or more times a year □ 

1 or 2 times a year □ 

Never □ 
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Student Item: 

23. About how often does a guest speaker come to speak to your science class? 

Teacher Item: 

About how often do you bring a guest speaker to talk to your science class? 



Both: 



3 or more times a year □ 

1 or 2 times a year □ 

Never or hardly ever □ 



Discrepancy rate 



9% (1 out of 1 1 responses) 



Number children reporting more frequently 0 
Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never or hardly ever 


1 -2 times/ year 


3 or more times/ year 


Never or hardly ever 


3 






1-2 times/ year 


1 


4 




3 or more times/ year 






3 



Discussion 

The results from this question are a good example of chance agreement. Even though their 
answers were the same, the students of one teacher were thinking of different guest speakers than the 
teacher. If both students and teachers had remembered all the guest speakers, the proper answer would 
have been “3 or more times a year” rather than the “one or two times a year” that four students and one 
teacher indicated. As for the previous question, the use of “Never or hardly ever” prevents the response 
options from being mutually exclusive. 



Recommendations 



The middle and last response options are not mutually exclusive. Change as follows: 



3 or more times a year □ 

1 or 2 times a year □ 

Never □ 



BEST COPY AVAILABLE 
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Teacher Item Only: 

Tl. During the last year, how much time in total have you spent in professional development 
workshops or seminars in science or science education? Include attendance at professional 
meetings and conferences, district-sponsored workshops, and external workshops. 



None □ 

Less than 6 hours □ 

6-15 hours □ 

16-35 hours □ 

More than 35 hours □ 



Teacher Responses 


Frequency 


None 




Less than 6 hours 




6-15 hours 




16-35 hours 


1 


More than 35 hours 


2 



Discussion 

The three teachers interviewed had spent at least 16 hours on professional development over the 
last year. Each had a different idea about what types of classes or training to include. One teacher, at 
first, did not include district workshops he did not find useful. When he changed his mind and decided to 
include them, his answer changed to “more than 35 hours.” Another teacher felt that “training in anything 
that would impact my science teaching” should be included. The third teacher was confused by the term 
“professional meetings” and included district steering committee meetings for science teaching. Some 
teachers interpreted “last year” to mean the past academic year, others interpreted it to mean the last 12 
months. Most of the teachers tried to find reasons to be inclusive — that is, to include activities rather 
than to exclude them from their estimates. 



Recommendations 

The time period to be included needs to be specified. The item already details the types of 
training to be included, but analysts should be aware that teachers will often err on the side of including 
marginally appropriate professional development activities in their estimates. 

During the past 12 months , how much time in total have you spent in professional development 
workshops or seminars in science or science education? Include attendance at professional 
meetings and conferences^ district-sponsored workshops , and external workshops. 
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Teacher Item Only: 

T2. How useful were any courses you took or any professional development you received in the last 
five years for each of the following areas? If you did not take any courses or receive 
professional development in the area, please check the first box (“No courses or professional 
development”). Please check one box per line. 

Cooperative group instruction? 



No courses or professional development □ 

Not useful □ 

Somewhat useful □ 

Moderately useful □ 

Very useful □ 



Teacher Responses 


Frequency 


No courses or professional 
development 




Not useful 


i 


Somewhat useful 


i 


Moderately useful 




Very useful 


i 



Discussion 

Again, teachers were inclusive in their answers. One teacher interpreted these items to include 
areas that were part of any course or part of any professional development in which he participated. 
Another teacher felt that professional development in cooperative group instruction was not useful 
because cooperative group instruction was not useful. “A student is a student.” Researchers need to take 
this into account when analyzing responses about areas whose utility is controversial. 

Teachers have generally similar views of the meaning of the response options. Following is a list 
of the teachers’ differing definitions of the four non-zero response options. The teachers use slightly 
different conceptual scales, and assign different increments to the ordinal scale. Overall, teachers were 
able to make meaningful distinctions between categories. These response options appear to work. 

Not useful 

♦ something that a teacher might have heard about but which was not applicable or relevant for 
their teaching 

♦ not being able to use it in class 

♦ training for which there was no need or impact 
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Somewhat useful 

♦ information that you think about and begin to apply in your teaching 

♦ okay - like a C student 

♦ training that was helpful but did not have the greatest impact 
Moderately useful 

♦ information that the teacher was able to use in the classroom and try a few times 

♦ something you can use 

♦ courses that were a bit more useful 

Very useful 

♦ something that was tried several times and that the teacher would like to continue employing 

♦ courses that really change things 

♦ something that can benefit everybody 



0 
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Teacher Item Only: 

T3. How useful were any courses you took or any professional development you received in the last 
five years for each of the following areas? If you did not take any courses or receive 
professional development in the area, please check the first box (“No courses or professional 
development”)- Please check one box per line. 

Interdisciplinary instruction? 



No courses or professional development □ 

Not useful □ 

Somewhat useful □ 

Moderately useful □ 

Very useful □ 



Teacher Responses 


Frequency 


No courses or professional 
development 




Not useful 




Somewhat useful 




Moderately useful 


2 


Very useful 


1 



Discussion 

There was no discussion of this item outside of a general discussion of items in this series. 
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Teacher Item Only: 

T4. How useful were any courses you took or any professional development you received in the last 
five years for each of the following areas? If you did not take any courses or receive 
professional development in the area, please check the first box (“No courses or professional 
development”). Please check one box per line. 

Teaching higher-order thinking skills? 



No courses or professional development □ 

Not useful □ 

Somewhat useful □ 

Moderately useful □ 

Very useful □ 



Teacher Responses 


Frequency 


No courses or professional 
development 




Not useful 




Somewhat useful 


1 


Moderately useful 




Very useful 


2 



Discussion 

One teacher lost the item’s context. The teacher forgot the stem (“How useful were any courses 
you took or any professional development you received in the last five years for each of the following 
areas?”). He did not take a course in teaching higher-order thinking skills, but still responded that it is 
very useful. 
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Recommendations 

Combining the question with the stem may help to remind teachers of the context of the answer: 

How useful were any courses you took or any professional development you received in the last 
five years for teaching higher-order thinking skills? Please check one box per line . 

I have not received any professional 
development or taken any courses in this 



area in the last five years □ 

Not useful □ 

Somewhat useful □ 

Moderately useful □ 

Very useful □ 
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Teacher Item Only: 

T5. How useful were any courses you took or any professional development you received in the last 
five years for each of the following areas? If you did not take any courses or receive 
professional development in the area, please check the first box (“No courses or professional 
development”). Please check one box per line. 

Teaching students from different cultural backgrounds? 



No courses or professional development □ 

Not useful □ 

Somewhat useful □ 

Moderately useful □ 

Very useful □ 



Teacher Responses 


Frequency 


No courses or professional 
development 


i 


Not useful 




Somewhat useful 


i 


Moderately useful 




Very useful 


i 



Discussion 

There was no discussion of this item outside of a general discussion of items in this series. 
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Teacher Item Only: 

T6. How useful were any courses you took or any professional development you received in the last 
five years for each of the following areas? If you did not take any courses or receive 
professional development in the area, please check the first box (“No courses or professional 
development”). Please check one box per line. 

Classroom management and organization? 



No courses or professional development □ 

Not useful □ 

Somewhat useful □ 

Moderately useful □ 

Very useful □ 



Teacher Responses 


Frequency 


No courses or professional 
development 




Not useful 


1 


Somewhat useful 


1 


Moderately useful 




Very useful 


1 



Discussion 

There was no discussion of this item outside of a general discussion of items in this series. 
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Teacher Item Only: 

T7. How useful were any courses you took or any professional development you received in the last 
five years for each of the following areas? If you did not take any courses or receive 
professional development in the area, please check the first box (“No courses or professional 
development”)- Please check one box per line. 

Other professional issues? 



No courses or professional development □ 

Not useful □ 

Somewhat useful □ 

Moderately useful □ 

Very useful □ 



Teacher Responses 


Frequency 


No courses or professional 
development 


i 


Not useful 




Somewhat useful 




Moderately useful 


i 


Very useful 


i 



Discussion 

One teacher was confused about this question; he was not sure what was meant by “other 
professional issues.” He answered “No courses or professional development.” Another responded with 
respect to a multimedia course supported by George Lucas' foundation. Still a third interpreted the item 
as including teaming, advisories, curricular issues, new standards, and school functions. 
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Recommendations 

The use of an “other” category will invariably result in idiosyncratic interpretations, making 
analysis of response frequencies difficult or impossible. Its value is that it does not compel the 
respondent to classify training in an ill-suited category. We suggest that a “specify” instruction be added 
to allow codebacks. 

How useful were any courses you took or any professional development you received in the last 
five years for each of the following areas? If you did not take any courses or receive 
professional development in the area, please check the first box (“No courses or professional 
development ”). Please check one box per line . 

Please specify any courses or training not included in the previous list: 
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Teacher Item Only : 

T8. Think about your plans for your science instruction during the entire year. About how much 
emphasis will you give to each of the following objectives for your students? Fill in one box on 
each line. 

Understanding key science concepts? 

Heavy emphasis □ 

Moderate emphasis □ 

Little or no emphasis □ 



Teacher Responses 


Frequency 


Heavy emphasis 


2 


Moderate emphasis 


1 


Little or no emphasis 





Discussion 

This item was interpreted by one teacher as referring to fundamental principles - a basic 
understanding of how things work. The cardiovascular system and its operation as a system was the 
example provided. Another defined the item as “concepts the school and the district say students have to 
learn before they leave class.” The periodic table and its meaning was an example that this teacher 
provided. 

Another teacher had a problem figuring out the context of items in this series. He initially 
answered about the emphasis he would give to each of these in planning lessons. Then, he thought the 
items were interested in finding out about in-class time. Finally, he went back to his initial interpretation. 



Recommendations 

This question is asking about a socially appropriate behaviors. It’s hard to imagine many teachers 
reporting that they do not give this a heavy emphasis. It should be noted that there may be a tendency for 
items in this series to elicit response of heavy or moderate emphasis due to social desirability factors. The 
teacher who reported “moderate emphasis” commented “it sounds like it should be given more 
emphasis.” 
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Teacher Item Only: 

T9. Think about your plans for your science instruction during the entire year. About how much 
emphasis will you give to each of the following objectives for your students? Fill in one box on 
each line. 

Developing science problem-solving skills? 

Heavy emphasis □ 

Moderate emphasis □ 

Little or no emphasis □ 



Teacher Responses 


Frequency 


Heavy emphasis 


3 


Moderate emphasis 




Little or no emphasis 





Discussion 

One teacher interpreted this item to include understanding science concepts, questioning, and 
observational skills. 
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Teacher Item Only: 

T10. Think about your plans for your science instruction during the entire year. About how much 
emphasis will you give to each of the following objectives for your students? Fill in one box on 
each line. 

Learning about the relevance of science to society and technology? 

Heavy emphasis □ 

Moderate emphasis □ 

Little or no emphasis □ 



Teacher Responses 


Frequency 


Heavy emphasis 


1 


Moderate emphasis 




Little or no emphasis 


1 


Missing 


i 



Discussion 

One teacher omitted this item on purpose. He said “the relevance of science to technology did 
not make a lot of sense. It seems as if this item should say “relevance of science and technology for 
society.” 



Recommendation 

To clarify this item, the following rewording should be considered: 
Learning about the relevance of science and technology to society? 

Heavy emphasis □ 

Moderate emphasis □ 

Little or no emphasis □ 
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Teacher Item Only: 

Til. Think about your plans for your science instruction during the entire year. About how much 
emphasis will you give to each of the following objectives for your students? Fill in one box on 
each line. 

Developing data analysis skills? 

Heavy emphasis □ 

Moderate emphasis □ 

Little or no emphasis □ 



Teacher Responses 


Frequency 


Heavy emphasis 


2 


Moderate emphasis 


1 


Little or no emphasis 





Discussion 

One teacher thought that “developing data analysis skills” referred to learning how to collect 
information, how to organize information, and what is learned from information. The teacher commented 
that “it sounds like it should be given more emphasis.” Again, social desirability may be a factor in the 
responses. 
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Teacher Item Only: 

T12. How often do you use each of the following to assess student progress in science? 

Individual projects or presentations? 



Once or twice a week □ 

Once or twice a month □ 

Once per grading period □ 

Once or twice a year □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Once or twice a week 




Once or twice a month 


2 


Once per grading period 


1 


Once or twice a year 




Never or hardly ever 





Discussion 

One teacher particularly liked the option with a grading-period anchor (“once per grading 
period”). He said it helped him to focus the linkage of the items to the grading period, and made it easier 
to respond. 

One teacher was confused about how to approach this and the next question. She has group 
projects but evaluates the individual students on their individual contributions. She answered both items 
in the same way, “to average things out.” 

Finally, one teacher lost context, forgetting the stem (“How often do you use each of the 
following to assess student progress in science”) and responded to the frequency of occurrence of the 
activities in items in this series. 
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Recommendations 

If the grading period option is kept (which seems reasonable), one should add an item asking 
about the number of grading periods per semester or per year. In addition, the low-frequency response 
options should then be linked to grading period, while the high frequency response options can remain 
unlinked to the grading period. In addition, context could be maintained by embedding the assessment 
construct in the option: 

Use individual projects or presentations to assess student progress in science? 



Once or twice a week □ 

Once or twice a month □ 

Two or three times per grading period □ 

Once per grading period □ 

Never □ 
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Teacher Item Only: 

T13. How often do you use each of the following to assess student progress in science? 

Group projects or presentations? 



Once or twice a week □ 

Once or twice a month □ 

Once per grading period □ 

Once or twice a year □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Once or twice a week 




Once or twice a month 


2 


Once per grading period 




Once or twice a year 




Never or hardly ever 


1 



Discussion 

There was no discussion of this item outside of a general discussion of the item. 



Recommendations 

Change option to reinstate context and change rating scale as in previous item. 
Use group projects or presentations to assess student progress in science? 



Once or twice a week □ 

Once or twice a month □ 

Two or three times per grading period □ 

Once per grading period □ 

Never □ 




The Measurement of Instructional Background Indicators 



202 



Eighth Grade Science Students and Teachers 



Teacher Item Only: 

T14. How often do you use each of the following to assess student progress in science? 

Laboratory notebooks or journals? 



Once or twice a week □ 

Once or twice a month □ 

Once per grading period □ 

Once or twice a year □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Once or twice a week 


1 


Once or twice a month 




Once per grading period 


2 


Once or twice a year 




Never or hardly ever 





Discussion 

There was no discussion of this item outside of a general discussion of the item. 



Recommendations 

Change option to reinstate context and change rating scale, as suggested previously. 
Use laboratory notebooks or journals to assess student progress in science? 



Once or twice a week □ 

Once or twice a month □ 

Two or three times per grading period □ 

Once per grading period □ 

Never □ 
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Teacher Item Only: 

T15. How often do you use each of the following to assess student progress in science? 

Hands-on activities? 



Once or twice a week □ 

Once or twice a month □ 

Once per grading period □ 

Once or twice a year □ 

Never or hardly ever □ 



Teacher Responses 


Frequency 


Once or twice a week 


1 


Once or twice a month 


2 


Once per grading period 




Once or twice a year 




Never or hardly ever 





Discussion 

One teacher included guest speakers doing hands-on activities with students. Another teacher 
responded that “the science lab teacher assesses this” but responded “once or twice a week,” estimating 
the lab teacher’s answer. The third teacher noted that these activities could be part of group projects or 
individual projects and activities. 

This raises another general issue: science labs. In some schools, science lab is a separate class; in 
others, a part of science class. Since both contribute to a student’s understanding of science (and, 
therefore, to their performance on NAEP achievement tests), information about whether the student is 
taking a separate lab course AND about instructional practices and activities in this course need to be 
obtained (through a survey of the students’ science lab teachers). 
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Recommendations 

Change option to reinstate context and change rating scale, as suggested previously. 
Use hands-on activities to assess student progress in science? 



Once or twice a week □ 

Once or twice a month □ 

Two or three times per grading period □ 

Once per grading period □ 

Never □ 
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Teacher Item Only: 

T16. How many computers are there for student use in your classroom? 



None available □ 

One within the classroom □ 

Two or three within the classroom □ 

Four or more within the classroom □ 



Teacher Responses 


Frequency 


None available 




One within the classroom 


i 


Two or three within the 
classroom 




Four or more within the 
classroom 


2 



Discussion 

The item seemed to work; validated through direct observation. 
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Teacher Item Only: 

T17. How difficult is it for your science students to access the computers in your school’s computer 
laboratory? 

There is no computer laboratory at my school □ 

It is easy for students to gain access to the computers 
in the computer laboratory □ 

It is difficult for students to gain access to the 
computers in the computer laboratory □ 



Teacher Responses 


Frequency 


There is no computer laboratory at my 
school 




It is easy for students to gain access to the 
computers in the computer laboratory 


2 


It is difficult for students to gain access to 
the computers in the computer laboratory 


1 



Discussion 

Two teachers had a hard time answering this question because it is easy for students to gain 
access to the computers in the computer lab before and after school, but it is difficult for students to gain 
access to the computers in the computer lab during the school day. One ended up answering in terms of 
access before and after school; another, in terms of use during the school day. A third teacher truly felt 
her answer would lie somewhere between “It is easy for students to gain access to the computers in the 
computer laboratory” and “It is difficult for students to gain access to the computers in the computer 
laboratory.” The response options for this question are too simplistic. 



Recommendations 

The purpose of this question needs to be more explicitly stated, and the response options 
expanded. 

How difficult is it for your students to use your schooVs computer laboratory: 



Very easy Easy 

a) During school hours? □ □ 

b) Outside of normal school hours? □ □ 



Very 

Difficult Difficult 

n n 

n n 
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Teacher Item Only: 

T18. How do you use computers for instruction in science? Fill in all boxes that apply. 



Play science/learning games to teach basic skills □ 

Play science/learning games to teach more 

advanced principles □ 

Providing access to computers as a reward for 

student performance □ 

Simulations and modeling □ 

Data analysis and other applications □ 

Word processing □ 

I do not use computers for science instruction □ 



Teacher Responses 


Frequency 


Play science/leaming games to teach basic 
skills 




Play science/leaming games to teach more 
advanced principles 


1 


Providing access to computers as a reward 
for student performance 


1 


Simulations and modeling 


2 


Data analysis and other applications 


2 


Word processing 


2 


I do not use computers for science 
instruction 


1 



Discussion 

One teacher checked “I do not use computers for instruction in science” and “Providing access to 
computers as a reward for student performance.” He interpreted the latter as allowing students to do 
homework, play games, or use the Internet on the computer if they had done well in science. He pointed 
out a clear formatting problem - the question states that respondents can “Fill in all boxes that apply.” 
However, the last option “I do not use computers for science instruction” was intended to be mutually 
exclusive to all the other options. If this option is picked along with other options, valid interpretations 
would be difficult. 

Another teacher suggested several additional options that could be added for the use of computer 
in instruction in science: research, CD-ROMs, Internet, productivity software (i.e., HyperCard, Hyper 
studio, webpage). Her students use HyperCard stacks and she wasn’t sure where this use would fall 
within the provided options, so she chose “simulations and modeling,” even though she said they do not 
actually use computers for simulations and modeling. 
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A third teacher interpreted “playing science/leaming games to teach basic skills” as asking about 
general education and learning skills. He said that computers are not ever used for this purpose in a 
science class. Only “Play science/leaming games to teach more advanced principles” was interpreted to 
include science class usage. 

Recommendations 

Other response options, particularly an Internet option should be considered. 

The item should be made into “Yes/No” questions to avoid issues associated with blanks. That 
is, an unchecked box cannot be unambiguously interpreted as meaning, “No.” It can also mean that the 
respondent didn’t know the answer, omitted the item (accidentally or deliberately), or didn’t understand 
the item. This would allow the final option (“I do not use computers for science instruction”) to be 
eliminated or changed to an “Other (SPECIFY)” option. 
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Teacher Item Only: 

T19. Which best describes the space where this class is taught? 



A classroom with no access to a laboratory or a water source □ 

A classroom with access to a water source only □ 

A classroom with access to a laboratory only □ 

A laboratory with water source □ 



Teacher Responses 


Frequency 


A classroom with no access to a laboratory 
or a water source 




A classroom with access to a water source 
only 


1 


A classroom with access to a laboratory 
only 


1 


A laboratory with water source 


1 



Discussion 

One problem with this question is that many science teachers use two rooms - a regular 
classroom and a lab. All the teachers had this problem. One teacher said that, since there is a science lab 
in other building, she checked “a classroom with access to a laboratory only.” This response could be 
misinterpreted without reference to her explanation. The third teacher also felt that his room was both a 
classroom and a lab. 



Recommendations 

We would suggest splitting this question into several questions: 



(a) Do you have access to a science lab? Yes □ No □ 

(b) Which best describes the space where this class is taught? 



A classroom with no access to a laboratory or a water source □ 

A regular classroom with access to a water source □ 

A regular classroom without access to a water source □ 

A classroom with access to a laboratory □ 

A laboratory with water source □ 
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Teacher Item Only: 

T20. As part of their work for this science class, do students produce any of the following records of 
their work? Fill in all boxes that apply. 



Notebooks or reports of laboratory work □ 

Reports or other written records of extended science projects □ 

Written reports on specific topics or issues in science □ 

Journals, diaries, or logs of ideas about science or work done for science class □ 

Three-dimensional scientific models □ 



Teacher Responses 


Frequency 


Notebooks or reports of laboratory work 


3 


Reports or other written records of 
extended science projects 


3 


Written reports on specific topics or issues 
in science 


1 


Journals, diaries, or logs of ideas about 
science or work done for science class 


3 


Three-dimensional scientific models 


3 



Discussion 

Due to time constraints, there was only a general discussion of this item. No obvious difficulties 
or problems were noted. 



O 
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Discussion of Eighth Grade Science Findings 

Discrepancy rates. Discrepancy rates for selected items are presented in table 8 (below). There 
were high discrepancy rates between student and teacher report, even for items for which within-class 
variation would be expected to be low. There are several reasons for these discrepancies — as well as for 
other discrepancies. They are discussed below. 



Table 8. Discrepancy Rate for Selected Eighth Grade Science Items 



When you study science in school, how often do you do each of the 
following: 


(n=H) 


Discuss science in the news 


18% 


Work with other students on a science activity or project 


82% 


Give an oral science report 


55% 


Do hands-on activities or investigations in science 


73% 


Talk about the measurements and results from your hands-on activities or 
investigations 


82% 


Go outside to observe or measure things 


36% 


Design and carry out your own science investigations 


55% 


When you study science in school, how often does your teacher do each of 
the following: 




Talk to the class about science 


27% 


Do a science demonstration 


45% 


Use computers for science (e.g., science software, telecommunications) 


18% 


About how often does your science class go on a science field trip? 


18% 


About how often does a guest speaker come to speak to your science 
class? 


9% 


Which best describes the science course you are taking? 


67% 


About how often do you study science in school? 


36% 


Do either you or your teacher save your science work in a portfolio? 


55% 


Do you ever do science projects in school that take a week or more? 


27% 


Which best describes the science course you are taking? 


67% 



NOTE: The number of students (n) is the modal number of respondents to each item. 



Nature of science instruction. Three factors related to the nature of science instruction 
contributed to these discrepancies: 

3) In the eighth grade, science instruction can include science laboratory activities. Without 
clear directions, many students will include these activities; many teachers will not, unless 
they are also teaching science labs. 

4) (Science instruction may not occur every day. When this is the case, the behavioral 
frequency response option “almost every day” can be interpreted literally (in which case, it 
would almost be never selected) or can be interpreted as meaning “almost every day we have 
science.” In the latter case, it would be selected more frequently. 

5) There are two different types of science homework. One is “regular” homework; the other 
refers to homework related to science projects. Items asking about homework might include 
regular or both types. 

Implicit and explicit time frame issues. The above example, interpreting “almost every day” as 
“almost every day we have science class,” is an example of an explicit time frame, because the item is 



The Measurement of Instructional Background Indicators 



212 



Eighth Grade Science Students and Teachers 



explicitly linked to the phrase, “When you study science in school.” However, the subsequent response 
options (“1 - 2 times a week” and “1 - 2 times a month”) implicitly link the behavior to the school year. 

If science is taught three times a week and the focal behavior occurs in two of these classes, the 
alternatives “Almost every day” and “1 - 2 times a week” can have nearly identical meanings to the 
respondent. 

There was a series of items asking students, “Have you ever done ...?” Although this item uses 
the word “ever,” it is not clear whether the literal interpretation is intended or the contextually derived 
interpretation (eighth grade, since most of the items ask about eighth grade experiences) is the proper one. 

Quite a few students would respond affirmatively because of experiences in other grades. 

Comprehension. Many of the terms and phrases employed in the items were not understood by 
all eighth graders. Some of the more problematic words and phrases were: 

♦ simple machines 

♦ levers 

♦ pulleys 

♦ content area 

♦ integrated science 

♦ portfolio 

♦ oral science report 

♦ investigations in science 

♦ measurements and results from your hands-on activities or investigations 

♦ design and carry out your own science investigation 

♦ e.g. 

♦ science software 

♦ telecommunications 

♦ teacher talk to the class about science 

♦ science demonstration 

Some of these items were also inconsistently interpreted by teachers. “Talking to the class about 
science” is an activity that teachers report they do every day. (The question seemed almost silly to some 
teachers, making them question their interpretation of it.) Teachers also had somewhat different ideas 
about what a science demonstration involved. 

Formatting issues. Two formatting issues are associated with high discrepancy rates. The first 
deals with items in a list format and is called loss of context. The second is a problem inherent in the use 
of a Check all that apply instruction. 

Loss of context. Many items are presented in a list format. For example, there is a series of items 
that begins with, “When you study science in school, how often does your teacher do each of the 
following?” This stem is followed by several items, including, “Use computers for science (e.g., science 
software, telecommunications).” Some people do not read the stem; others, as they go through the list, 
forget the stem. At least two students reported about their own use of computers, rather than their 
teacher’s. Although their responses matched their teachers, due to coincidental agreement, this type of 
problem occurred frequently, in several different series of items. 
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Check all that apply items. An example of such an item is: 

Have you ever done hands-on activities or projects in school with any of the following? Fill in all 
boxes that apply. 



If a student does not understand the question, they will often leave it blank. Accordingly, when 
these data are analyzed, one cannot be certain how a blank box should be interpreted. It can mean “No,” 
“I don’t understand the question,” or “I don’t know if I have done this.” 

Scale issues. Two rating scale problems were noted: overlapping response categories and gaps 
between response categories were noted. In the following scale, there are overlapping response 
categories: 

About how often does a guest speaker come to speak in your science class? 

3 or more times a year □ 

1 or 2 times a year □ 

Never or hardly ever □ 

If this happened once, either “Never or hardly ever” or “1 or 2 times a year” could be checked. 

As noted in the general findings, gaps between response categories refers to the fact that 
behaviors performed with certain frequencies (such as three times a week) fall between two adjacent 
response categories (“once or twice a week” and “once or twice a month”). Respondents will develop 
their own idiosyncratic rules for categorizing such behaviors. 



Magnifying glass or microscope (for looking at small things)? 
Thermometer or barometer (for making measurements)? 
Simple machines (for example, pulleys and levers)? 



□ 

□ 

□ 
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Student Item: 

1. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Do mathematics problems from textbooks 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 

Do mathematics problems from textbooks 



Both: 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



65% (13 out of 20 responses) 



Number children reporting more frequently 5 

Number children reporting less frequently 8 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 










1-2 times/ week 






3 


5 


Almost every day 


2 




6 


4 



Discussion 

One source of discrepancy was the term “from.” One teacher took textbook problems and 
prepared them on sheets and in other formats. He did not feel that his students would know the source of 
these problems. This probably contributed to many of the cases of student underreporting. 

Another teacher reported that her students do problems from their textbooks almost every day. 
However, she felt that the students would not realize this since their “textbook” is actually a series of 
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small books. She thought they would consider a reference-like book they used as their textbook. So, she 
responded in terms of the frequency with which her students do problems from their reference-like 
textbook: 1 - 2 times per week. However, five (out of eight) of her students seemed to consider the series 
of small books as their textbook and reported “Almost every day.” 

The third teacher pointed out a problem with the phrase “students in your class. ” She 
commented that mathematics problems from textbooks are begun in school, but are typically finished at 
home. From the wording of the item (“How often do students in your class do each of the following ; ”) 
there is no guidance as to whether this means “your students” or whether it refers to what the students do 
when they are physically in the classroom. This teacher responded “every day.” However, all of her 
students responded less frequently. 

Additionally, one student had a question about the time period of interest. She asked, “Do they 
want to know about this week, this year, or what?” She tried to answer the question by averaging over 
the course of the year. 



Recommendations 

Respondents can generally provide more accurate behavioral frequency data for a recent, short, 
well-defined period of interest (i.e., “last week”) than for an ambiguously defined time period. Serious 
consideration should be given to asking these items about the last week of classes. If not, the term 
“generally” can be used as a qualifier (i.e., “.. how often do you generally do each of the following ...”) 



If these items are intended to get at the frequency of classroom behaviors (and we are assuming 
that this is their intent), the wording of the stem should be changed to: 

Teacher Questionnaire: How often do your students do each of the following when they are in 
your classroom? 

Student Questionnaire: When you do mathematics in your classroom, how often do you do 
each of the following? 

It seems unrealistic to expect students to know the source of problems that teachers provide on 
sheets or in other formats. They can probably respond more accurately to a simpler item: 

Do mathematics problems from books 
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Student Item: 

2. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Do mathematics problems on worksheets? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



60% (14 out of 20 responses) 



Number children reporting more frequently 10 

Number children reporting less frequently 4 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 


2 


2 


2 


2 


1-2 times/ week 




1 




6 


Almost every day 






1 


4 



Discussion 

One teacher assigns worksheets for homework. However, she reported that her students only do 
worksheets in school “1-2 times/month.” Four of her students included homework, and reported a more 
frequent use of worksheets. Her students who reported “never or hardly ever” were responding to the 
current unit, for which few worksheets were employed. In other words, there is a time frame issue. That 
is, because the time frame is not specified, some students answered about the immediate past while the 
teacher used a semester-long time frame. 

Students also seemed to have a more inclusive definition of worksheet than their teachers do. 
Many students considered any handout with problems to be a worksheet. Teachers more narrowly define 
this term. 
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Recommendations 

The term worksheet has very different meanings to teachers and students. It does not appear to be 
an item that can be easily defined for students. Accordingly, it is a candidate for elimination. 

The issues of restricting this activity to in-class activity and specifying a period of interest, as 
discussed in the previous item are also relevant here. 
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Student Item: 

3. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Solve mathematics problems with a partner or in small groups? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



55% (11 out of 20 responses) 



Number children reporting more frequently 0 

Number children reporting less frequently 1 1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 










1-2 times/ week 


3 


3 


1 




Almost every day 


2 


1 


2 


8 



Discussion 

Teachers included informal grouping and problem solving in their responses. At least five 
children in one class (for which the teacher reported this happening 1-2 times/week) underreported the 
frequency of this activity because they were not considering informal discussions. 

In one class, seating was organized to encourage small group interaction in problem solving. In 
this class, one student who underreported the frequency of this behavior (in comparison with his teacher) 
explained that he and his classmates do not work together and that in the group setting they each solve 
their own problems. Other cases of student underreporting reflect the fact that some students responded 
to only formal activities. 
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Recommendations 

We suggest a total restructuring of this item, to better assess the construct of interest: 

In your classroom, how frequently do your students work in small groups or with a single 
partner? 



Nearly all of the time □ 

Most of the time □ 

About half of the time □ 

Some of the time □ 

Never or almost never □ 



This item can be made into a series of items that asks about the proportion of time devoted to (a) 
whole class instruction, (b) large group instruction, and individual instruction, as well. 
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Student Item: 

4. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Work with measuring instruments or geometric solids? 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 

T22. Work with objects like rulers? 

T23. Work with counting blocks or geometric shapes? 



Both: 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



55% (1 1 out of 20 responses) 



Number children reporting more frequently 4 

Number children reporting less frequently 7 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


8 


4 






1 -2 times/ month 










1-2 times/ week 










Almost every day 


2 


2 


3 


1 



NOTE: Teacher responses to items T22 and T23 were combined into a single teacher response. 



Discussion 

The phrase “work with objects like rulers” creates a logical problem in that it attempts to define a 
class of objects with only a single example. The salient feature(s) of “ruler” will determine how the item 
is interpreted. One teacher thought it was referring to things that are used to draw, measure, and/or 
produce numerical outcomes. She included protractors, compasses, and dice (since they produced 
numerical outcomes) as examples of objects like rulers. Another teacher interpreted this phrase as 
referring to devices used to make straight lines. The third interpreted it as referring to measuring devices, 
but also included compasses as an example. 
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The use of the term “geometric solids” was confusing to students. Most knew that it referred to 
three-dimensional objects. However, at least six of the 20 students did not know what it meant. One 
student thought a protractor was a geometric solid. This term, besides being incomprehensible to many, 
was also irrelevant for most. Although these students use geometric shapes frequently, they never (or 
almost never) use geometric solids. The term “measuring instruments” was usually, but not always, 
interpreted by students as including rulers. At least one student stated that she did NOT think of rulers 
when answering. 

There are issues associated with rare events and whether something that has happened 2-3 times 
should be categorized as “Never or hardly ever” or “Once or twice a month.” Such issues contributed to 
the discrepancies. Other discrepancies were associated with the unanchored time frame — that is, some 
students were considering the most recent week while others were considering the entire semester. 



Recommendations 

The phrase “work with objects like rulers” in the teacher item should be changed to “work with 
measuring instruments like rulers and protractors ” (if this is the intent of the item.) 

The student item should be changed to ’’work with rulers, measuring instruments, and geometric 
shapes.” 

The issue of specifying a period of interest, as discussed previously, with a different set of 
categories should help improve the accuracy of reporting. 
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Student Item: 

5. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Write a few sentences about how you solved a mathematics problem? 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 
Write a few sentences about how to solve a mathematics problem? 

Both: 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



50% (9 out of 18 responses) (2 missing) 
3 
6 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


2 




2 


1 


1-2 times/ month 










1-2 times/ week 


2 


2 


3 




Almost every day 


1 




1 


4 



Discussion 

One of the teachers reported that her students “never or hardly ever” wrote a few sentences about 
how to solve a mathematics problem. However, one of her students said “every day,” because he does 
this on his own to help him remember how he did it. Another of her students included answers that had 
words in them. 

At least three of the students lost context. That is, they forgot the stem (“When you do 
mathematics in school, how often ...?”). They interpreted it as asking them to “Write a few sentences 
about how to solve a mathematics problem.” One student thought it was a trick question, since he 
couldn’t possibly write a sentence in the small box provided! 
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Recommendations 

For students, the item can be reworded to ask, “How often does your teacher make you write a 
few sentences that explain how you solve a mathematics problem?” 

For items in a series, it is essential that context be provided and maintained. This is a general 
issue and also applicable to items on the Teacher’s Questionnaire. 
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Student Item: 

6. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Take mathematics tests? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



25% (5 out of 20 responses) 



Number children reporting more frequently 5 

Number children reporting less frequently 0 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 




15 


4 


1 


1-2 times/ week 










Almost every day 











Discussion 

Teachers were asked if they included standardized tests in their estimates. They all responded 
negatively. They are responding to this item as asking about tests which they and/or their department 
develop and administer. Since the incidence of standardized tests is low, their exclusion does not appear 
to be a major issue, for teachers. 

Students were more likely to include standardized tests. This created a discrepancy when one 
student responded with respect to the most recent week, during which standardized testing occurred on 
several days. The other four discrepancies were all associated with one class, for which students included 
weekly quizzes. (The teacher only reported the major tests.) 
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Recommendations 

To avoid underreporting of quizzes, the item can be changed to ask about “tests and quizzes.” 
Conversely, if it is important to distinguish between tests and quizzes, separate items can be asked about 
each. 
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Student Item: 

7. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Talk to the class about your mathematics work? 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 

Talk to the class about their mathematics work? 



Both: 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



80% (16 out of 20 responses) 



Number children reporting more frequently 7 

Number children reporting less frequently 9 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 


2 


2 


2 


2 


1-2 times/ week 


5 


2 


2 


3 


Almost every day 











Discussion 

Two of the three teachers lost context -- that is, they forgot that the item was asking about how 
often their students talked to the class about their mathematics work. They initially responded with 
respect to their own behavior. (The responses presented in the table reflect their best estimates of how 
frequently their students engaged in the desired behavior.) 

Even when teachers understood that the item was referring to their students, there were still major 
differences in item interpretation. One teacher felt that this required the student to be in front of the class, 
engaging the other students in dialogue. She explained that students are at the board daily, solving 
problems. However, “they are not talking to the class ~ they are demonstrating what they did.” 
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Another teacher thought only of formal student presentations of their work. During these 
presentations, the students describe their response process to the class. There are only a few students who 
can do this, and therefore there are only a few students who are asked to do this, since the teacher does 
not like to force students to do things. So, her reported frequency was neither an average nor reflective of 
the typical student. 

The third respondent included situations when students talked to individual students or small 
groups of students about their work. 

Students’ interpretations of these items were similarly diverse: 

♦ “How often do you raise your hand to give the answer to the question when the teacher is 
looking for volunteers?” (at least 2 students said this or made similar comments) 

♦ How often you go to the board to show how you solved a homework problem. 

♦ Several believed that the item required you to stand in front of the class. 

♦ Talking to your friends about your math grades. (There is textual support for this 
interpretation.) 

♦ Informal conversations with your friends. 

There is a general issue: Whether the teacher is responding to the performance of this behavior 
by ANY student or by the typical student. The item can be interpreted either way. 

Recommendations 

Even with adults, it is important that context be maintained in list-formatted items. Adding a 
little redundancy by changing the item to: “Students talk to the class about their mathematics work” 
should eliminate this problem. 

The intent of this item needs to be specified to allow an effective item to be developed. If the 
intent is to get at formal presentations, an item asking about how often “Students speak to the entire class 
about their mathematics work” can be considered. However, it is not clear whether this wording will 
elicit situations where a student is called upon to answer a question. If the intent is to get at extended 
presentations, the following wording can be considered for teachers: “Students provide a detailed and in- 
depth description to the entire class about how they solved a mathematics problem” 

Similar wordings can be developed for students. 

A decision must be made about whether the item is referring to any student or the typical student, 
so that the stem can be appropriately modified. 
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Student Item: 

8. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Do 10 or more practice problems in mathematics by yourself? 

Teacher Validation Probe (not a survey item): 

How often do your students do 10 or more practice problems in mathematics, by themselves, in 
class? 



Both: 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 

Number children reporting more frequently 
Number children reporting less frequently 



38% (5 out of 13 responses) (7 missing) 
4 
1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


4 


1 


2 


1 


1-2 times/ month 










1-2 times/ week 










Almost every day 


1 






4 
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Discussion 

Data for only 13 students are presented since the validation probe was not administered to one 

teacher. 

The term “practice problems” was confusing. Interpretations included: 

♦ “extra problems to practice what you’re doing in class” 

♦ work sheet problems 

♦ practice for the SATs 

♦ “how often you do mathematics problems on your own time just to get better at math” 

♦ homework 

♦ “problems we make up ourselves” 

♦ homework and classwork combined 

♦ “warm-up exercises before a test” 

♦ “problems to practice what we went over in class” 

♦ “problems in class where it is not a test” 

At least one student lost context and reported “almost every day” because he was considering 
practice problems he does at home. 

Recommendations 

As previously noted, restoration of context (i.e., that this item is asking about classroom 
behaviors) is necessary. 

The adjective “practice” confuses rather than enhances. Since another item asks about 
worksheets, “practice problems” seems to refer to another type of problem. But, we do not understand 
what type of problem is the focus of this item. If a clearer definition cannot be provided, the item should 
be deleted. 
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Student Item: 

9. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Discuss solutions to mathematics problems with other students? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



60% (12 out of 20 responses) 



Number children reporting more frequently 1 

Number children reporting less frequently 1 1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 










1-2 times/ week 


4 




2 


1 


Almost every day 


2 


1 


4 


6 



Discussion 

This is another item for which the teacher could respond about the performance of this behavior 
by ANY student or by the typical student. The item wording provides textual support for both of these 
interpretations. The fact that 1 1 out of the 12 discrepancies were associated with student underreporting 
is suggestive of the fact that teachers were responding about ANY student. 

Teachers typically included both formal and informal discussions. However, many students only 
considered formal discussions. At least two students thought it was asking about formal discussions of 
solutions to the problems such as the POW (problem of the week.) Other students felt this item was 
really the same as item 3 (“Solve mathematics problems with a partner or in small groups”) and/or item 7 
(“Talk to the class about your mathematics work”). Several other students thought it was asking about 
how often they help other students with their work. 
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Recommendations 

A decision must be made about whether the item is referring to any student or the typical student, 
so that the stem can be appropriately modified. 

If this item is intended to capture informal discussions in class, this point must be made explicit 
to the students. Wording such as: “Talk with other students about how you solved a math problem ” 
might be able to effectively convey such a concept. 
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Student Item: 

10. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Use a computer? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



45% (9 out of 20 responses) 



Number children reporting more frequently 5 

Number children reporting less frequently 4 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1 -2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 


10 


2 




3 


1-2 times/ month 










1-2 times/ week 










Almost every day 


4 






1 



Discussion 

One teacher interpreted this as asking about any use of a computer in class. For this class, it was 
rare, since computers were not used until about two weeks prior to our discussions. 

However, one student responded about the last 30 days, in which the computer was used fairly 

often. 



Other students responded about computer usage at home since they lost context. In other words, 
in response to the stimulus, “Use a computer,” they responded to their use of a computer anywhere and 
for any purpose. 

Another teacher interpreted the item as asking about any use of a computer, by any student, at 
home, in other classes, and in this class. There is textual support for this interpretation, especially since 
there is no restriction to math activities. 
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Recommendations 



As mentioned previously, the time period of interest needs to be made more explicit in the item 



stem. 



As noted earlier, it is important that context be maintained in list-formatted items. Adding a little 
redundancy by changing the item to: “Use a computer in mathematics class ” should eliminate this 
problem. 

A decision must be made about whether the item is referring to any student or the typical student, 
so that the stem can be appropriately modified. 
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Student Item: 

11. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Use a calculator? 

Teacher Validation Probe (not a survey item): 

About how often do your students use a calculator to do their math class work? 



Both: 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



20% (4 out of 20 responses) 



Number children reporting more frequently 4 

Number children reporting less frequently 0 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


5 




1 


1 


1-2 times/ month 










1-2 times/ week 






3 


2 


Almost every day 








8 



Discussion 

One teacher prohibited the use of calculators in his classroom and responded “Never or hardly 
ever.” However, one of his students reported about homework and home usage of the calculator (because 
she lost context) and another admitted to using the calculator in class in spite of the teacher’s prohibition. 

In another class, the teacher permitted classroom usage. The typical student used calculators 1 - 2 
times/week. However, a few (two students) used it more frequently. 




The Measurement of Instructional Background Indicators 

243 



235 



Eighth Grade Mathematics Students and Teachers 



Recommendations 

As mentioned previously, it is important that context be maintained in list-formatted items. 
Adding a little redundancy by changing the item to: “Use a calculator in mathematics class ” should 
eliminate this problem. 

A decision must be made about whether the item is referring to any student or the typical student, 
so that the stem can be appropriately modified. 




The Measurement of Instructional Background Indicators 



236 



244 



Eighth Grade Mathematics Students and Teachers 



Student Item: 

12. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Write reports or do mathematics projects? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



20% (4 out of 20 responses) 



Number children reporting more frequently 3 

Number children reporting less frequently 1 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


’ 4 




1 




1-2 times/ month 


i 


12 


2 




1-2 times/ week 










Almost every day 











Discussion 

Two students lost context and responded to how often they write reports in school. They 
included reports for subjects other than mathematics, leading to overreporting. Other errors were errors 
of estimation. For instance, the student who replied “Never or hardly ever” was able to recall three 
instances of mathematics projects. Her teacher categorized this as 1 - 2 times/month, as did most of the 
students in her class. (There may have been more than three such projects.) 

Recommendations 

To maintain context, the item can be changed to: “Write reports for math class or do math 
projects . ” This should eliminate the problem. 
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Student Item: 

13. When you do mathematics in school, how often do you do each of the following? Fill in only 
one box on each line. 

Teacher Item: 

How often do the students in your class do each of the following? Fill in one box on each line. 



Both: 



Work and discuss mathematics problems that reflect real-life situations? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



65% (13 out of 20 responses) 



Number children reporting more frequently 5 

Number children reporting less frequently 8 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1 -2 times/ 
week 


Almost every 
day 


Never or hardly ever 










1-2 times/ month 


3 


4 


5 




1-2 times/ week 










Almost every day 




3 


2 


3 



Discussion 

The phrase “reflect real-life situations” was interpreted by teachers as meaning: 

♦ Events taken from the news. None of her students interpreted the phrase in this way. Any 
agreement with her response of 1 - 2 times/month was coincidental. 

♦ Problems based on something that students are interested in, such as roller skating or walking 
rates 

♦ Situations that would reflect what students consider to be everyday applications of algebra 

Some students’ interpretation of the phrase “real-life situations” was more inclusive than their 
teacher’s. Other students recalled specific real-life problems (the Problem of the Week) and responded 
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with respect to the frequency of that specific problem. On further probing, they could recall other 
problems but they simply responded to the most salient type of real-life problem. 

At least three students did not know what was meant by “real-life situations.” One student 
thought that it was career related. 

Recommendations 

The phrase “real-life situations” can be interpreted as either referring to things that occur in the 
real world or situations that are relevant to the life of an eighth grade. If the interest is in the latter, one 
can ask teachers about "... problems that deal with situations that are of interest and relevance to your 
students . ” This wording would need to be changed if it were to be asked of eighth graders. 
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Student Item: 

14. Do either you or your teacher have a portfolio with your mathematics work in it? 

Teacher Validation Probe (not a survey item): 

Do you or your students have portfolios for their math work? 



Both: 



Yes □ 
No □ 



Discrepancy rate 



30% (6 out of 20 responses) 



Teacher Responses 


Student responses 


No 


Yes 


No 


5 


2 


Yes 


4 


9 



Discussion 

Some discrepancies arose when students kept their work in a file folder of their own. Since 
“portfolio” is not an eighth grade word, at least two students interpreted portfolio to mean any folder, 
causing a discrepancy. Another student thought portfolio meant “a binder with your work in it.” So, she 
responded negatively while her teacher responded positively. 

In other classes, there are school-wide portfolios for which math work can be contributed. In this 
class, the teacher responded positively but some students responded negatively since neither they nor their 
teacher has such a portfolio. (The school does.) 

Recommendations 

To eliminate confusion about whether the student, the teacher, or the school keeps a portfolio, the 
item should focus on one. This can be done by asking, “Does your teacher have a portfolio or folder for 
every student in your class, in which he or she keeps their best work?” It would be simpler to ask the 
teacher this directly. 
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Student Item: 

15. Do you have a calculator that you can use to do mathematics schoolwork? 

Teacher Validation Probe (not a survey item): 

Do all of your students have calculators? 



Both: 



Yes □ 
No □ 



Discrepancy rate 



45% (9 out of 20 responses) 



Teacher Responses 


Student responses 


No 


Yes 


No 


3 


9 


Yes 




8 



Discussion 

In spite of the discrepancy rate, this item is functioning as intended. A teacher’s response that all 
of his or her students do not have calculators means that some of their students may have a calculator. In 
fact, such is clearly the case. 



Recommendations 

Leave as is. 
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Student Item: 

16. For mathematics class, how often do you use a calculator for each of the following activities? 
Fill in one box on each line. 

Classwork? 

Teacher Validation Probe (not a survey item): 

About how often do your students use a calculator to do their math class work? 



Both: 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Discrepancy rate 



35% (7 out of 20 responses) 



Number children reporting more frequently 4 

Number children reporting less frequently 3 



Teacher 

Responses 


Student Responses 


Never or hardly 
ever 


1-2 times/ month 


1-2 times/ 
week 


Almost every 
day 


Never or hardly ever 


6 






1 


1-2 times/ month 










1-2 times/ week 


1 


1 




3 


Almost every day 






1 


7 



Discussion 

Due to within class variation in calculator usage, students’ answers should not be expected to 
exactly match their teacher’s responses. However, the item had two problems: 

1) Two students lost the context of the question. They forgot that the question was asking about 
their use of calculators for math and answered with respect to how often they do math 
classwork. 

2) One student included homework in his response. He felt that classwork and homework were 
the same thing. 
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Recommendations 

To deal with the loss of context and the misinterpretation of classwork, we propose a focusing of 
the item: 

How often do you use a calculator in your math class? 
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Student Item: 

17. For mathematics class, how often do you use a calculator for each of the following activities? 
Fill in one box on each line. 

Homework? 

Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 

Teacher Validation Probe (not a survey item): 

Do your students use calculators for their math homework? 

Yes □ 

No □ 



Discrepancy rate 



25% (5 out of 20 responses) 



Teacher 

Responses 


Student Responses 


Never or 
hardly ever 


Yes 


1 -2 times/ 
month 


1 -2 times/ 
week 


Almost 
every day 


No 


2 


5 


Yes 




13 



Discussion 

Two students lost the context of the question. They forgot that the question was asking about 
their use of calculators for math and answered with respect to how often they do math homework. 



Recommendations 

Again, to deal with the loss of context, we suggest a context-reinstating phrase be added: 
How often do you use a calculator for your math homework? 
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Student Item: 

18. For mathematics class, how often do you use a calculator for each of the following activities? 
Fill in one box on each line. 

Tests or quizzes? 



Almost every day □ 

Once or twice a week □ 

Once or twice a month □ 

Never or hardly ever □ 



Teacher Validation Probes (not survey items): 

Are students ever allowed to use calculators for math tests? What are your rules about 
the use of calculators on math tests? Can they use them for all math tests? 

Yes □ 

No □ 



Discrepancy rate 



15% (3 out of 20 responses) 



Teacher 

Responses 


Student Responses 


Never or 
hardly ever 


Yes 


1-2 times/ 
month 


1-2 times/ 
week 


Almost 
every day 


No 


10 


2 


Yes 


1 


7 



Discussion 

Two students lost the context of the question. They forgot that the question was asking about their 
use of calculators for math tests and answered with respect to how often they take math tests or quizzes. 

One student indicated that it depends on the unit, and since in his mind the use of calculators for 
tests is most often not allowed, he said “never or hardly ever.” However, the teacher’s response was 
“yes.” 



Recommendations 

Similar to the two other questions in this series, the item should be reworded to avoid the loss of 

context: 



How often do you use a calculator for math tests or quizzes? 
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Student Item: 

19. What kind of mathematics class are you taking this year? 



I am not taking mathematics this year □ 

Eighth-grade mathematics □ 

Prealgebra □ 

Algebra □ 

Integrated or sequential mathematics □ 

Applied mathematics (technical preparation) □ 

Other mathematics class □ 



Teacher Validation Probe (not a survey item): 

What kind of mathematics class is this? 



Eighth-grade mathematics □ 

Prealgebra □ 

Algebra □ 

Integrated or sequential mathematics □ 

Applied mathematics (technical preparation) □ 

Other mathematics class □ 



Discrepancy rate 



23% (4 out of 13 responses; 7 missing) 



Teacher 

Responses 


Student Responses 


1 am not 
taking math 
this year 


8 th 

grade 

math 


Pre- 

algebra 


Algebra 


Integrate 
d math 


Applied 

math 


Other 

math 

class 


Checked 

two 

boxes or 
more 


I am not taking 
math this year 


















8 th grade math 




5 












3 


Pre-algebra 


















Algebra 






1 


4 










Integrated math 


















Applied math 


















Other math 
class 
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Discussion 

This question was very hard for eighth graders to answer. They either did not understand parts of 
the question, did not know how to complete it, or did not know the answer, as described below. 

At least 15 students (75%) did not know what “integrated or sequential” math meant. Here are a 
few definitions: 

♦ Difficult stuff mixed together 

♦ Advanced or high school math 

♦ Different combinations of math or grade levels 

In addition, at least 13 students did not know the meaning of “applied math (technical 
preparation).” One student defined it as “preparation for technology,” and another as “a course you apply 
for, to get to college.” Finally, one student did not know what “algebra” meant and another student was 
not sure what was meant by “other mathematics class.” 

One teacher was not asked the validation probe, which is the reason that seven student responses 
are missing. Two of these seven students checked two boxes. So, a total of five students (25%) checked 
two boxes or more. Four of the five students checked both “eight-grade mathematics” and “pre-algebra.” 
One student felt that these two responses were the same thing. Another student indicated that the question 
does not specify to only check one, which is a valid point. One student explained that he does prealgebra 
at home, which was the reason for his checking the two boxes. Since the question does not specify “in 
school,” this appears to be a legitimate interpretation. Finally, one student checked three boxes (“eight- 
grade mathematics,” “prealgebra,” and “integrated or sequential mathematics”). He thought integrated 
and sequential math should be separated into two categories. 

Three students either did not know the name of their math class or guessed. They all responded 
“eighth-grade mathematics.” 

Recommendations 

Asking eighth graders what type of math class they are taking appears to be too hard for them to 
answer. It would be better to ask their teachers. If the question is administered to eighth grade students, 
it should be revised: the response options should be simplified and it should be specified that only one 
option can be checked. Also, none of the students responded “I am not taking mathematics this year,” so 
this option might not be needed for eighth graders. 

What kind of mathematics class are you taking this year? Check only one box . 

Eighth-grade mathematics □ 

Prealgebra □ 

Algebra □ 

Other mathematics class □ 
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Student Item: 

20. About how much time do you usually spend each day on mathematics homework? 



I am not taking mathematics this year □ 

None □ 

15 minutes □ 

30 minutes □ 

45 minutes □ 

1 hour □ 

More than 1 hour □ 



Teacher Item: 

Approximately how much mathematics homework do you assign to students in this class each 
day? 



None □ 

15 minutes □ 

30 minutes □ 

45 minutes □ 

One hour □ 

More than one hour □ 



Discrepancy rate 65% (13 out of 20 responses) 

Number children reporting more frequently 4 



Number children reporting less frequently 7 



Teacher 

Responses 


Student Responses 


1 am not 
taking math, 
this year 


None 


15 

minutes 


30 

minutes 


45 

minutes 


1 

hour 


More 
than 1 
hour 


Two 

boxes 

checked 


I am not taking 
math, this year 


















None 


















15 minutes 


















30 minutes 






2 


4 








2 


45 minutes 






2 


3 


3 


3 


1 




1 hour 


















More than 1 
hour 
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Discussion 

Most of the discrepancies in this question were due to the fact that time spent on math homework 
varies from student to student. However, there were problems associated with the item. 

♦ Two students checked two boxes. For example, one student checked “15 minutes” and “30 
minutes” because she usually spends about 15-30 minutes per day on math homework. 

♦ One teacher included regular homework and “long-term homework” (project work) in her 
response. However, two of her students did not include project work in their answers, which 
led to discrepancies. 

♦ One teacher included work assigned that students might complete in class. In other words, 
she included homework students might complete either in school or at home. 

♦ One student lost context, answering with respect to how much time she spends on homework 
each night, as opposed to just math homework. 



Recommendations 

It should be specified that only one box should be checked. Also, none of the students responded 
“I am not taking mathematics this year,” so this option might not be needed for eighth graders. 

About how much time do you spend on math homework each day? Check only one box. 



None □ 

15 minutes □ 

30 minutes □ 

45 minutes □ 

1 hour □ 



More than one hour □ 
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Student Item Only: 

21. How much do you agree with each of the following statements? Fill in one box on each line. 
I am good at mathematics. 



Strongly agree □ 

Agree □ 

Undecided □ 

Disagree □ 



Strongly disagree □ 



Student Responses 


Frequency 


Strongly agree 


2 


Agree 


11 


Undecided 


6 


Disagree 


1 


Strongly disagree 


0 



Discussion 

One student indicated that he did not know what it meant to be good at mathematics and 
responded “undecided.” Another student said “agree,” though he thought it was a strange question 
because he was not sure if it meant good at math or good in math class. He thought those were different 
things. Finally, one student reported that she was “undecided” and explained: “It all depends... Math is 
such a broad statement. There is calculus and then there is 2+2.” 



Recommendations 

None. 
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Student Item Only: 

22. How much do you agree with each of the following statements? Fill in one box on each line. 

There is only one correct way to solve a mathematics problem. 



Strongly agree □ 

Agree □ 

Undecided □ 

Disagree □ 

Strongly disagree □ 



Student Responses 


Frequency 


Strongly agree 




Agree 




Undecided 


2 


Disagree 


11 


Strongly disagree 


7 



Discussion 

Two students chose “undecided” because they thought it was true that for some math problems 
there is only one correct way to solve them, whereas for other math problems this is not true. Thus, they 
had trouble thinking abstractly. One of these students would have responded positively to the statement 
that “Some mathematics problems can be solved in many different ways.” Another student was unclear 
on what was being asked. 



Recommendations 

None. 
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Student Item Only: 

23. How much do you agree with each of the following statements? Fill in one box on each line. 

Describing mathematical concepts and ideas is as important as doing mathematical 
operations such as addition and multiplication in solving problems. 



Strongly agree □ 

Agree □ 

Undecided □ 

Disagree □ 



Strongly disagree □ 



Student Responses 


Frequency 


Strongly agree 


2 


Agree 


5 


Undecided 


10 


Disagree 


2 


Strongly disagree 


0 


Skipped 


1 



Discussion 

This item was very hard for eighth graders to understand. In fact, seven of the students who 
responded “undecided” chose this option because they did not understand the meaning of the question. 
Another student was completely confused by the question and skipped it altogether. Even students who 
were able to provide a response other than “undecided” made comments like: 

♦ This is kind of hard to explain 

♦ I have trouble understanding it 

♦ It is too long and confusing 

♦ It is sort of a mouthful 

The phrase “describing mathematical concepts” seemed to create particular difficulties for the 
students. Below are some of their definitions of this term: 

♦ Ways of doing concepts 

♦ Adding and multiplying 

♦ What you did and how you solved it 

♦ An equation you wanted to solve 
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Similar to the previous question, two of the students chose “undecided” because they thought 
describing math concepts and doing math operations are both important. Thus, again they had trouble 
thinking abstractly. 

There is a further logical problem with this item. If a student feels that “describing mathematical 
concepts and ideas” is much more or much less important than “doing mathematical operations,” (s)he 
will answer “disagree.” Thus, disagreement does not inform about whether the student feels concepts and 
ideas are more or less important than operations - only that they are not equally important. 



Recommendations 

This question is too abstract, long, and confusing for most eighth graders to understand. It should 
be drastically simplified or deleted. Consider: 

It is more important to learn how to multiply and divide than to learn the theory of math. 



The Measurement of Instructional Background Indicators 



261 



253 



Eighth Grade Mathematics Students and Teachers 



Teacher Item Only : 

Tl. During the last year, how much time in total have you spent in professional development 
workshops or seminars in mathematics or mathematics education? Include attendance at 
professional meetings and conferences, district-sponsored workshops, and external 
workshops. 



None □ 

Less than 6 hours □ 

6-15 hours □ 

16-35 hours □ 

More than 35 hours □ 



Responses 


Frequency 


None 




Less than 6 hours 




6-15 hours 




16-35 hours 


i 


More than 35 hours 


2 



Discussion 

The phrase during the last year was interpreted by two teachers as referring to the school year and 
by one as referring to the last 12 months. 

There is a general tendency for teachers to want to report engaging in positive behaviors, such as 
attending professional development workshops. One of the respondents, a first-year teacher, considered 
Math Department meetings to be professional development. Although such experiences would not be 
considered professional development by most teachers, the respondent felt that he learned quite a lot 
about mathematics teaching in these meetings. 



Recommendations 

Depending on the intent of the item, “During the last year” should be changed to either “Since the 
end of the last school year” or “During the last 12 months.” 

To reduce the likelihood of teachers reporting about Math Department meetings, the item could 
be decomposed and separate questions asked about (a) professional meetings and conferences, (b) district- 
sponsored workshops, and (c) external workshops. However, the increased burden associated with three 
items rather than only one might preclude such an approach. 
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Teacher Items Only: 

T2-7. How useful were any courses you took or any professional development you received in the 
last five years for each of the following areas? If you did not take any courses or receive 
professional development in the area, please check the first box (“No courses or 



professional development” 


'). Please check one box per line. 






2. Cooperative group instruction 


No courses or 
professional 
development 
□ 


Not 

Useful 

□ 


Somewhat 

Useful 

□ 


Moderately 

Useful 

□ 


Very 

Useful 

□ 


3. Interdisciplinary instruction 


□ 


□ 


□ 


□ 


□ 


4. Teaching higher-order thinking skills 


□ 


□ 


□ 


□ 


□ 


5. Teaching students from different 
cultural backgrounds 


□ 


□ 


□ 


□ 


□ 


6. Classroom management and 
organization 


□ 


□ 


□ 


□ 


□ 


7. Other professional issues 


□ 


□ 


□ 


□ 


□ 


Responses 












2. Cooperative group instruction 


No courses or 
professional 
development 


Not 

Useful 


Somewhat 

Useful 

2 


Moderately 

Useful 


Very 

Useful 

1 


3. Interdisciplinary instruction 


1 


- 


- 


2 


- 


4. Teaching higher-order thinking skills 


- 


- 


- 


1 


2 


5. Teaching students from different 
cultural backgrounds 


- 


- 


1 


- 


2 


6. Classroom management and 
organization 


1 


- 


1 


1 


- 


7. Other professional issues 


- 


- 


- 


2 


1 
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Discussion 

General comments. Respondents had little trouble using the rating scale and recognized that the 
rating scale ordered items in terms of usefulness. Usefulness was correctly interpreted in the context of 
having an impact on pedagogical practices. 

There were several issues that emerged that are important in analysis of findings: 

♦ One of the teachers was a first-year teacher. For all “new” teachers (i.e., those with fewer 
than four or fewer years’ experience), courses taken will include undergraduate courses. For 
first year teachers, nearly all of the responses will be about their undergraduate courses. 

♦ There is a clear tendency to overreport engagement in these activities. As one respondent 
pointed out, teachers want people to think they are involved. So, teachers would look for 
reasons to indicate that they had received each type of training. Receiving any professional 
development is a loosely defined construct. It includes things that are parts of courses or 
parts of other training received. For example, teaching higher order thinking skills was part 
of training that was received in gifted and talented education training. So, the teacher 
responded affirmatively to item 4, even though this training was not very extensive. 

♦ Related to a desire to report engagement in these activities is “telescoping” — that is, 
reporting experiences that occurred outside of the period of interest. One teacher reported 
taking a course but, on further probing, admitted that it may well have occurred outside of the 
five-year period of interest. 

♦ Evaluating training with respect to its utility is potentially problematic. That is, excellent 
training might not be reported as being useful if the teacher is already quite proficient so that 
the training would have little or no impact on teaching practices. 

It should be noted that these items were modified from the way they were asked in the 1995-96 
Teacher Questionnaire. The 1995-96 questionnaire merely asked whether the teacher had taken courses 
or participated in professional development activities in each of the listed areas. This shift in focus still 
provides information about whether or not a course was taken or professional development was received 
in an area, and amplifies the value of this information with an assessment of its perceived value. 

Specific comments. Item 3 asks about interdisciplinary instruction. One respondent felt very 
strongly that this was not relevant for eighth grade mathematics — especially for algebra. This might be a 
valid point. However, this teacher also was very proud of her “voting project.” For this project, students 
use actual election data to determine the percent of people who actually vote. Clearly, such a project can 
be labeled interdisciplinary instruction. 

Item 5 (Teaching students from different cultural backgrounds) was responded to affirmatively by 
a teacher who had taken Spanish language courses. Although this may not be what was intended by the 
item writers, it is clearly an appropriate example of a course that was taken that was relevant to “teaching 
students from different cultural backgrounds.” In fact, two of the three teachers considered foreign 
language courses to be relevant and appropriate. 

Item 6 (Classroom management and organization) was interpreted by one participant to include 
Internet training. She defined classroom organization as “the way groups are organized to learn things.” 
Since some groups were organized to learn things through the Internet, she felt this was appropriate. This 
is an example of teachers interpreting terms as inclusively as possible. 

Item 7 (Other professional issues) was a difficult item to answer. One teacher immediately 
commented that she did not know what this meant. With probing, she was able to come up with “Test 
Development” and responded to this. Another teacher thought of teaching differentially abled students. 
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The third teacher answered with respect to a planning grant for professional development that supported 
her partially for a year. Given the diversity of areas responded to, this is a difficult item to interpret. 



Recommendations 

The addition of a usefulness scale to these items was an attempt to assess the quality of the 
training received. It appears that respondents can provide such information, but this information is 
difficult to interpret. 

Certain of the items in the list of areas are problematic. “Teaching students from different 
cultural backgrounds” is often interpreted to include foreign language courses. It is not clear that this is 
what is desired -- although knowledge of a foreign language can be invaluable for this type of instruction. 
If this item is intended to get at pedagogical techniques, the phrase “excluding foreign language 
courses M must be added. Alternatively, if the item is intended to include foreign language courses, it may 
be working. To be sure of this, one should include “ Foreign language classes ” as an explicit listed item. 
The use of this option, preceding “Teaching students from different cultural backgrounds” should obviate 
the need for an “excluding foreign language courses ” instruction. 

“Other professional issues” is another problematic item. Given the diversity of interpretation, it 
should either be eliminated or a “( SPECIFY)” instruction should be added to aid interpretation. 
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Teacher Item Only: 

T8-9. What level of exposure, if any, have you had to each of the following topics or areas? Fill 
in all boxes that apply on each line and at least one box on each line. 





One or More 


Part of a 


Professional 






College or 


College or 


Development 






University 


University 


Workshops or 


Little or No 




Courses 


Course 


Seminars 


Exposure 


Methods of teaching elementary 
mathematics 


□ 


□ 


n 


□ 


College algebra 


□ 


□ 


□ 


□ 



Responses 





One or More 


Part of a 


Professional 






College or 


College or 


Development 






University 


University 


Workshops or 


Little or No 




Courses 


Course 


Seminars 


Exposure 


8. Methods of teaching elementary 










mathematics 


1 


- 


1 


- 


NOTE: One respondent checked the first three 


boxes. 




9. College algebra 


2 


- 


- 


1 



Discussion 

One of the teachers, who almost certainly had some exposure in school to elementary 
mathematics teaching methods, only checked “Professional Development Workshops or Seminars.” She 
started to check the first box and changed her mind. It seems that “top-down processing” came into play, 
leading her to choose only one box per line 1 . Unfortunately, our questionnaire did not use bold font to 
emphasize this point (as was done in the 1995-96 NAEP questionnaire). This formatting would, 
hopefully, be sufficient to ameliorate the problem. 

One teacher commented that the option “Professional Development Workshops or Seminars” did 
not make sense for “College algebra.” 



Recommendations 

The “College algebra” option should be removed from this grid and changed into an item that 
asks how many college algebra courses the respondent has taken. 



1 Top-down processing refers to the respondent’s adoption of a questionnaire response production heuristic (e.g., “Check only 
one box per line”) and its application for all subsequent items. 
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Teacher Item Only: 

T10-14. Have you ever studied any of the following, either in college or university courses or in 
professional development workshops or seminars? Fill in one box on each line. 





Yes 


No 


10. Use of manipulatives (e.g., counting blocks 
or geometric shapes) in mathematics instruction 


□ 


□ 


1 1 . Use of calculators in mathematics instruction 


□ 


□ 


12. Understanding students’ thinking about 
mathematics 


□ 


□ 


13. Gender issues in the teaching of mathematics 


□ 


□ 


14. Teaching students from different cultural backgrounds 


□ 


□ 


Responses 


Yes 


No 


10. Use of manipulatives (e.g., counting blocks 
or geometric shapes) in mathematics instruction 


3 


- 


11. Use of calculators in mathematics instructions 


- 




12. Understanding students’ thinking about 
mathematics 


3 


- 


13. Gender issues in the teaching of mathematics2 


1 




14. Teaching students from different cultural backgrounds 


3 


- 



Discussion 

All of these items were answered affirmatively by the three respondents, with the exception of 
“Gender issues in the teaching of mathematics.” One respondent indicated that she never attended a 
workshop or seminar totally devoted to this issue, and responded negatively. This is somewhat unusual, 
given the fact that teachers generally are over-inclusive in their reporting of their training experiences. 

The negative response may reflect a desire by the respondent to introduce some variance in her responses. 

As an example of being over-inclusive, another respondent answered affirmatively to having 
studied “Gender issues in the teaching of mathematics” because of its peripheral inclusion in some 
course, workshop, or seminar that she attended. 

Some of the courses, workshops, and seminars were in the distant past. A teacher included 
professional development and courses from 1970 and 1985. However, “ever” means “ever” and the 
teacher’s response reflects an appropriate interpretation of the question. 
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Finally, one participant validly commented that Question 14 (asking if the teacher had ever 
studied “Teaching students from different cultural backgrounds”) sounded like Question 5 (asking about 
the usefulness of courses on “Teaching students from different cultural backgrounds”). 



Recommendations 

Teachers tended to be over-inclusive in their reporting of their training experiences, presenting 
themselves as positively as possible. All of the items were answered affirmatively, with the exception of 
one teacher’s response to one of the five items. Therefore, the items may not have much discriminating 
power. One might consider limiting the time frame to the past few years, or increasing the specificity of 
the topics. 
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Teacher Item Only: 

T15. How many computers are there for student use in your classroom? 

□ None available 

□ One within the classroom 

□ Two or three within the classroom 

□ Four or more within the classroom 



Responses 


Frequency 


None available 


1 


One within the 
classroom 


2 


Two or three within 
the classroom 




Four or more within 
the classroom 





Discussion 

The teachers appeared to have no trouble responding to this item. Responses agreed with the 
interviewers’ visual inspection of the classroom. 



Recommendations 

None. 
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Teacher Item Only: 

T16. How difficult is it for your mathematics students to access the computers in your school’s 
computer laboratory? 

□ There is no computer laboratory at my school. 

□ It is easy for students to gain access to the computers in the computer laboratory. 

□ It is difficult for students to gain access to the computers in the computer laboratory. 



Responses 


Frequency 


There is no computer laboratory at 
my school. 




It is easy for students to gain 
access to the computers in the 
computer laboratory. 


3 


It is difficult for students to gain 
access to the computers in the 
computer laboratory. 





Discussion 

This is a new item. The respondents described access issues (consent forms are required by the 
district for students to be allowed to use the Internet). The original item, which was part of the number of 
computers item, provided options saying “Available in a computer laboratory (but difficult/and easy) to 
access and schedule.” 

One respondent asked, “Does this mean my students? Does it mean during my class time or at 
other times?” She decided it meant at any time. Another respondent raised an interesting point. While it 
is easy for individual students to gain access to the computer lab, it is very hard to sign up for a three-day 
block of time to bring her entire class there. 



Recommendations 

This item could be more sharply focused, asking about the ease with which a student can use 
computers in the computer laboratory whenever the student wants to do so. 
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Teacher Item Only: 

T17. How do you use computers for instruction in mathematics? Fill in all boxes that apply. 

□ Playing mathematics/leaming games to teach basic skills 

□ Playing mathematics/leaming games to teach more advanced principles 

□ Providing access to computers as a reward for student performance 

□ Simulations and modeling 

□ Data analysis and other applications 

□ Word processing 

□ I do not use computers for mathematics instruction 



Responses 


Frequency 


Playing mathematics/leaming 
games to teach basic skills 




Playing mathematics/leaming 
games to teach more advanced 
principles 


1 


Providing access to computers as 
a reward for student performance 


1 


Simulations and modeling 


1 


Data analysis and other 
applications 


2 


Word processing 




I do not use computers for 
mathematics instruction 


1 



Discussion 

One teacher felt that the graphing calculators that her students used were computers, so she 
responded affirmatively to “Simulations and modeling.” “Simulations and modeling” were a source of 
confusion for another respondent. She thought of programs that mimic flying or driving (i.e., Flight 
Simulator). This respondent also checked off “Data analysis and other applications,” since she had used 
software for graphing purposes. However, “Data analysis and other applications” evoked a comment 
from another teacher. He indicated that this was not really applicable for his class. Finally, a respondent 
asked, “How do you use word processing for instruction in mathematics?” This is a valid question, 
pointing out a flaw with the option. 
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Recommendations 

The term “Simulation and modeling” was not understood by all respondents. This term could be 
clarified by providing examples. 

Although the teachers appeared to consider each option, the absence of an “x” can indicate that 
the respondent did not know the answer, did not understand the option, or did not use computers in this 
way. It is therefore preferable to format this type of item as a series of “yes/no” questions rather than as 
a “check all that apply” item. 
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Teacher Item Only: 

T31. Do you permit students in this class unrestricted use of calculators? 



□ Yes 

□ No 



Responses 


Frequency 


Yes 


i 


No 


2 



Discussion 

One teacher answered “Yes,” indicating that students are allowed to use calculators at any time. 
However, there are restrictions, for behavioral reasons, since some students were playing too many games 
with their calculators. The teacher recognized the implicit meaning of the question. 



Recommendations 

None. 
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Teacher Item Only: 

T32. Do you permit students in this class to use calculators for tests? 



□ Yes 

□ No 



Responses 


Frequency 


Yes 


2 


No 


1 



Discussion 

One of the teachers who answered “Yes,” indicated that there was one exception: there was one 
question on one test for which students were not allowed to use a calculator. The other teacher who 
answered affirmatively wrote on the survey: “Tests done @ home!” to explain why she permits this. She 
can not prohibit it, so she allows it. 



Recommendations 

Some teachers include tests done at home in their responses. Consider specifying “in-class tests” 
if this is the intent of the item: 

Do you permit students in this class to use calculators for your in-class tests? 
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Teacher Item Only: 



T33-35. How well prepared do you feel to teach each of the following in this class? Fill in one 
box on each line. 





Very 

Well 

Prepared 


Moderately 

Well 

Prepared 


Not Very 

Well 

Prepared 


Not at All 
Prepared 


33. Mathematical concepts 


□ 


□ 


□ 


□ 


34. Mathematical procedures 


□ 


□ 


□ 


□ 


35. Computers 


□ 


□ 


□ 


□ 


Responses 


Very 

Well 

Prepared 


Moderately 

Well 

Prepared 


Not Very 

Well 

Prepared 


Not at All 
Prepared 


33. Mathematical concepts 


3 


- 


- 


- 


34. Mathematical procedures 


3 


- 


- 


- 


35. Computers 


- 


3 


- 


- 



Discussion 

One respondent defined “very well prepared,” as someone who can explain anything; 

“moderately well prepared,” as someone who needs some help; “not very well prepared,” as someone 
who needs more help in teaching; and “not at all prepared,” as someone with no knowledge or experience. 
Another respondent defined “very well prepared” as “thought through and practiced”; “moderately well 
prepared,” as a teacher who has done some of the above things, but not all; “not very well prepared,” as 
someone who spent about five minutes of preparation time for the class; and “not very well prepared” 
elicited the response, “I don’t know what that means.” This response option was so far removed from her 
experiential base as to be meaningless. 

One teacher felt that there was not much difference between Question 33 (Mathematical 
concepts) and Question 34 (Mathematical procedures). Nonetheless, she was able to define them. 
Concepts were defined as “overarching, big ideas.” Procedures were smaller concerns. For example, 
“data analysis” was a concept. Building histograms and making stem-and-leaf plots were procedures. 

In answering Question 35 (Computers), one teacher responded to how well prepared she felt to 
use graphing calculators, since she considered them to be a type of computer. 
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Recommendations 

Consider deleting “not at all prepared.” One teacher could not envision this phrase being 
applicable to herself. Furthermore, none of the teachers provided either this option or the proximal “not 
very well prepared” option as a response. 

There is also a technical problem with “not very well prepared” as a response option, which 
occurs when its complement, “very well prepared” also appears as a response option in the same item. 
Literally, any response option that is not “very well prepared” (e.g., “moderately well prepared” and “not 
at all prepared”) can be considered to be “not very well prepared.” That is, “very well prepared” and “not 
very well prepared” form a dichotomy, encompassing all responses. If this option is retained, it should be 
reworded to “not well prepared.” 



The Measurement of Instructional Background Indicators 



268 



276 



Eighth Grade Mathematics Students and Teachers 



Discussion of Eighth Grade Mathematics Findings 

Discrepancy rates. Discrepancy rates for selected items are presented in table 9 (below). As 
with other subjects and other grade levels, there were high discrepancy rates between student and teacher 
report, even for items for which within-class variation would be expected to be low. Reasons for these 
and other discrepancies are discussed below. 



Table 9. Discrepancy Rate for Selected Eighth Grade Mathematics Items 



When you do mathematics in school, how often do you do each of the 
following: 


(n=20) 


Do mathematics problems from textbooks 


65% 


Do mathematics problems on worksheets 


60% 


Solve mathematics problems with a partner or in small groups 


55% 


Work with measuring instruments or geometric solids 


55% 


Write a few sentences about how you solved a mathematics problem 


50% 


Take mathematics tests 


25% 


Talk to the class about your mathematics work 


80% 


Do 10 or more practice problems by yourself 


38% 


Discuss solutions to mathematics problems with other students 


60% 


Use a computer 


45% 


Use a calculator 


20% 


Write reports or do mathematics projects 


20% 


Work and discuss mathematics problems that reflect real-life situations 


65% 


Do either you or your teacher have a portfolio with your mathematics work 
in it? 


30% 


What kind of mathematics class are you taking this year? 


23% 



NOTE: The number of students (n) is the modal number of respondents to each item. 



Comprehension. Many of the terms and phrases employed in the items were not understood by 
all eighth graders. Some of the more problematic words and phrases were: 

♦ geometric solids 

♦ practice problems 

♦ problems that reflect real-life situations 

♦ portfolio 

♦ integrated or sequential mathematics 

♦ applied mathematics (technical preparation) 

Teachers would interpret tests in a slightly different way from their students. In responding to 
items about tests, teachers would not think about (and therefore would exclude) externally mandated tests. 
Students considered these externally mandated tests to be mathematics tests. 

Homework. Mathematics classwork includes assignments that are begun in school. Some 
students complete these assignments during class. Those who don’t complete the assignments in class 
have to finish them as homework. The assignments, as well as homework intended to be completed 
entirely outside of the classroom, might be subsequently discussed in class. Accordingly, items dealing 
with classwork sometimes would include homework behaviors. 
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Loss of context. Many items are presented in a list format. For example, there is a series of 
items that begins: 

When you do mathematics in school, how often do you do each of the following? 

♦ Write a few sentences about how to solve a mathematics problem 

♦ Use a computer 

♦ Write reports or do mathematics projects 

Students would lose context and answer about their overall computer usage in school and their overall 
frequency of report writing. One student was particularly confused by “Write a few sentences about how 
to solve a mathematics problem,” and thought she was being asked to write a few sentences about how to 
solve a math problem! 

Similarly, there was loss of context for the question: 

For mathematics class, how often do you use a calculator for each of the following activities? 

♦ Classwork 

♦ Homework 

♦ Tests or quizzes 

At least two students lost the calculator context and responded about the frequency of classwork, 
homework, and tests or quizzes. 
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Marilyn Se astro m 
Arnold Goldstein 



Jeffrey Owings 

Jeffrey Owings 

Jeffrey Owings 

Jeffrey Owings 

Samuel Peng 
Samuel Peng 

Jeffrey Owings 

Ralph Lee 

Jeffrey Owings 



Steven Kaufman 
Dawn Nelson 
Dawn Nelson 
Aurora D’Amico 
Ralph Lee 
Marilyn Se astro m 



National Household Education Survey (NHES) 

95- 12 Rural Education Data User’s Guide 

96- 13 Estimation of Response Bias in the NHES:95 Adult Education Survey 

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult 
Education Component 

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96-21 1993 National Household Education Survey (NHES:93) Questionnaires: Screener, School 

Readiness, and School Safety and Discipline 

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 



Samuel Peng 
Steven Kaufman 
Steven Kaufman 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 



Title 



NCES contact 



No. 

96-29 

96- 30 

97- 02 

97-03 

97-04 

97-05 

97-06 

97-08 

97-19 

97-20 

97-25 

97-28 

97-34 

97-35 

97-38 

97-39 

97- 40 

98- 03 
98-10 

2002-04 



Undercoverage Bias in Estimates of Characteristics of Adults and 0- to 2-Year-Olds in the 
1995 National Household Education Survey (NHES:95) 

Comparison of Estimates from the 1995 National Household Education Survey 
(NHES:95) 

Telephone Coverage Bias and Recorded Interviews in the 1993 National Household 
Education Survey (NHES:93) 

1991 and 1995 National Household Education Survey Questionnaires: NHES:91 Screener, 
NHES:91 Adult Education, NHES:95 Basic Screener, and NHES:95 Adult Education 
Design, Data Collection, Monitoring, Interview Administration Time, and Data Editing in 
the 1993 National Household Education Survey (NHES:93) 

Unit and Item Response, Weighting, and Imputation Procedures in the 1993 National 
Household Education Survey (NHES:93) 

Unit and Item Response, Weighting, and Imputation Procedures in the 1995 National 
Household Education Survey (NHES:95) 

Design, Data Collection, Interview Timing, and Data Editing in the 1995 National 
Household Education Survey 

National Household Education Survey of 1995: Adult Education Course Coding Manual 
National Household Education Survey of 1995: Adult Education Course Code Merge 
Files User’s Guide 

1996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 
Comparison of Estimates in the 1996 National Household Education Survey 
Comparison of Estimates from the 1993 National Household Education Survey 
Design, Data Collection, Interview Administration Time, and Data Editing in the 1996 
National Household Education Survey 

Reinterview Results for the Parent and Youth Components of the 1996 National 
Household Education Survey 

Undercoverage Bias in Estimates of Characteristics of Households and Adults in the 1996 
National Household Education Survey 

Unit and Item Response Rates, Weighting, and Imputation Procedures in the 1996 
National Household Education Survey 

Adult Education in the 1990s: A Report on the 1991 National Household Education 
Survey 

Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 

Improving Consistency of Response Categories Across NCES Surveys 



Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Peter Stowe 
Peter Stowe 

Kathryn Chandler 

Kathryn Chandler 
Kathryn Chandler 
Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Kathryn Chandler 

Peter Stowe 

Peter Stowe 

Marilyn Seastrom 



National Longitudinal Study of the High School Class of 1972 (NLS-72) 

95-12 Rural Education Data User’s Guide Samuel Peng 

2002-04 Improving Consistency of Response Categories Across NCES Surveys Marilyn Seastrom 



National Postsecondary Student Aid Study (NPSAS) 

96-17 National Postsecondary Student Aid Study: 1996 Field Test Methodology Report 
2000-17 National Postsecondary Student Aid Study: 2000 Field Test Methodology Report 
2002-03 National Postsecondary Student Aid Study, 1999-2000 (NPSAS: 2000), CATI 
Nonresponse Bias Analysis Report. 

2002-04 Improving Consistency of Response Categories Across NCES Surveys 



Andrew G. Malizio 
Andrew G. Malizio 
Andrew Malizio 

Marilyn Seastrom 



National Study of Postsecondary Faculty (NSOPF) 

97- 26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

98- 15 Development of a Prototype System for Accessing Linked NCES Data 

2000-01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

2002-04 Improving Consistency of Response Categories Across NCES Surveys 



Linda Zimbler 
Steven Kaufman 
Linda Zimbler 
Marilyn Seastrom 



Postsecondary Education Descriptive Analysis Reports (PEDAR) 

2000-1 1 Financial Aid Profile of Graduate Students in Science and Engineering Aurora D’Amico 



Private School Universe Survey (PSS) 

95-16 Intersurvey Consistency in NCES Private School Surveys Steven Kaufman 

95-17 Estimates of Expenditures for Private K-12 Schools Stephen Broughman 



Title 



NCES contact 



No. 

96-16 

96-26 

96- 27 

97- 07 

97- 22 

98- 15 
2000-04 

2000-15 



Strategies for Collecting Finance Data from Private Schools 
Improving the Coverage of Private Elementary-Secondary Schools 
Intersurvey Consistency in NCES Private School Surveys for 1993-94 
The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 
Schools: An Exploratory Analysis 

Collection of Private School Finance Data: Development of a Questionnaire 
Development of a Prototype System for Accessing Linked NCES Data 
Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 

Feasibility Report: School-Level Finance Pretest, Private School Questionnaire 



Recent College Graduates (RCG) 

98-1 5 Development of a Prototype System for Accessing Linked NCES Data 
2002-04 Improving Consistency of Response Categories Across NCES Surveys 



Schools and Staffing Survey (SASS) 

94-01 Schools and Staffing Survey (SASS) Papers Presented at Meetings of the American 
Statistical Association 

94-02 Generalized Variance Estimate for Schools and Staffing Survey (SASS) 

94-03 1991 Schools and Staffing Survey (SASS) Reinterview Response Variance Report 

94-04 The Accuracy of Teachers’ Self-reports on their Postsecondary Education: Teacher 

Transcript Study, Schools and Staffing Survey 

94- 06 Six Papers on Teachers from the 1990-91 Schools and Staffing Survey and Other Related 

Surveys 

95- 01 Schools and Staffing Survey: 1994 Papers Presented at the 1994 Meeting of the American 

Statistical Association 

95-02 QED Estimates of the 1990-91 Schools and Staffing Survey: Deriving and Comparing 
QED School Estimates with CCD Estimates 

95-03 Schools and Staffing Survey: 1990-91 SASS Cross-Questionnaire Analysis 

95-08 CCD Adjustment to the 1990-91 SASS: A Comparison of Estimates 

95-09 The Results of the 1993 Teacher List Validation Study (TLVS) 

95-10 The Results of the 1991-92 Teacher Follow-up Survey (TFS) Reinterview and Extensive 

Reconciliation 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

95-12 Rural Education Data User’s Guide 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

95-15 Classroom Instructional Processes: A Review of Existing Measurement Approaches and 
Their Applicability for the Teacher Follow-up Survey 
95-16 Intersurvey Consistency in NCES Private School Surveys 

95- 1 8 An Agenda for Research on Teachers and Schools: Revisiting NCES’ Schools and 

Staffing Survey 

96- 01 Methodological Issues in the Study of Teachers’ Careers: Critical Features of a Truly 

Longitudinal Study 

96-02 Schools and Staffing Survey (SASS): 1995 Selected papers presented at the 1995 Meeting 
of the American Statistical Association 

96-05 Cognitive Research on the Teacher Listing Form for the Schools and Staffing Survey 

96-06 The Schools and Staffing Survey (SASS) for 1998-99: Design Recommendations to 
Inform Broad Education Policy 

96-07 Should SASS Measure Instructional Processes and Teacher Effectiveness? 

96-09 Making Data Relevant for Policy Discussions: Redesigning the School Administrator 

Questionnaire for the 1998-99 SASS 

96-10 1998-99 Schools and Staffing Survey: Issues Related to Survey Depth 

96-1 1 Towards an Organizational Database on America’s Schools: A Proposal for the Future of 
SASS, with comments on School Reform, Governance, and Finance 
96-1 2 Predictors of Retention, Transfer, and Attrition of Special and General Education 
Teachers: Data from the 1989 Teacher Followup Survey 
96-1 5 Nested Structures: District-Level Data in the Schools and Staffing Survey 
96-23 Linking Student Data to SASS: Why, When, How 
96-24 National Assessments of Teacher Quality 




Stephen Broughman 
Steven Kaufman 
Steven Kaufman 
Stephen Broughman 

Stephen Broughman 
Steven Kaufman 
Dan Kasprzyk 

Stephen Broughman 



Steven Kaufman 
Marilyn Seastrom 



Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 

Sharon Bobbitt & 
John Ralph 
Samuel Peng 
Samuel Peng 

Sharon Bobbitt 

Steven Kaufman 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 

Dan Kasprzyk 

Dan Kasprzyk 
Dan Kasprzyk 
Dan Kasprzyk 
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No. 

96-25 

96- 28 

97- 01 

97-07 

97-09 

97-10 

97-11 

97-12 

97-14 

97-18 

97-22 

97-23 

97-41 

97-42 

97- 44 

98- 01 
98-02 
98-04 
98-05 

98-08 

98-12 

98-13 

98-14 

98-15 

98-16 

1999-02 

1999-04 

1999-07 

1999-08 

1999-10 

1999-12 

1999-13 

1999-14 

1999- 17 

2000- 04 

2000-10 

2000-13 

2000-18 

2002-04 



Title 

Measures of Inservice Professional Development: Suggested Items for the 1998-1999 
Schools and Staffing Survey 

Student Learning, Teaching Quality, and Professional Development: Theoretical 

Linkages, Current Measurement, and Recommendations for Future Data Collection 
Selected Papers on Education Surveys: Papers Presented at the 1996 Meeting of the 
American Statistical Association 

The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 
Schools: An Exploratory Analysis 
Status of Data on Crime and Violence in Schools: Final Report 

Report of Cognitive Research on the Public and Private School Teacher Questionnaires 
for the Schools and Staffing Survey 1993-94 School Year 
International Comparisons of Inservice Professional Development 
Measuring School Reform: Recommendations for Future SASS Data Collection 
Optimal Choice of Periodicities for the Schools and Staffing Survey: Modeling and 
Analysis 

Improving the Mail Return Rates of SASS Surveys: A Review of the Literature 
Collection of Private School Finance Data: Development of a Questionnaire 
Further Cognitive Research on the Schools and Staffing Survey (SASS) Teacher Listing 
Form 

Selected Papers on the Schools and Staffing Survey: Papers Presented at the 1997 Meeting 
of the American Statistical Association 

Improving the Measurement of Staffing Resources at the School Level: The Development 
of Recommendations for NCES for the Schools and Staffing Survey (SASS) 
Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 
State Assessments and State NAEP, Feasibility Study 
Collection of Public School Expenditure Data: Development of a Questionnaire 
Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report 
Geographic Variations in Public Schools’ Costs 

SASS Documentation: 1993-94 SASS Student Sampling Problems; Solutions for 

Determining the Numerators for the SASS Private School (3B) Second-Stage Factors 
The Redesign of the Schools and Staffing Survey for 1999-2000: A Position Paper 
A Bootstrap Variance Estimator for Systematic PPS Sampling 
Response Variance in the 1994-95 Teacher Follow-up Survey 
Variance Estimation of Imputed Survey Data 

Development of a Prototype System for Accessing Linked NCES Data 
A Feasibility Study of Longitudinal Design for Schools and Staffing Survey 
Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 
Measuring Teacher Qualifications 

Collection of Resource and Expenditure Data on the Schools and Staffing Survey 
Measuring Classroom Instructional Processes: Using Survey and Case Study Fieldtest 
Results to Improve Item Construction 
What Users Say About Schools and Staffing Survey Publications 
1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume III: Public-Use 
Codebook 

1993- 94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 
Indian Affairs (BIA) Restricted-Use Codebook 

1994- 95 Teacher Followup Survey: Data File User’s Manual, Restricted-Use Codebook 
Secondary Use of the Schools and Staffing Survey Data 

Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 

A Research Agenda for the 1999-2000 Schools and Staffing Survey 
Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core of 
Data (CCD) 

Feasibility Report: School-Level Finance Pretest, Public School District Questionnaire 
Improving Consistency of Response Categories Across NCES Surveys 



NCES contact 

Dan Kasprzyk 

Mary Rollefson 

Dan Kasprzyk 

Stephen Broughman 

Lee Hoffman 
Dan Kasprzyk 

Dan Kasprzyk 
Mary Rollefson 
Steven Kaufman 

Steven Kaufman 
Stephen Broughman 
Dan Kasprzyk 

Steve Kaufman 

Mary Rollefson 

Michael Ross 

Stephen Broughman 
Steven Kaufman 
William J. Fowler, Jr. 
Steven Kaufman 

Dan Kasprzyk 
Steven Kaufman 
Steven Kaufman 
Steven Kaufman 
Steven Kaufman 
Stephen Broughman 
Dan Kasprzyk 
Dan Kasprzyk 
Stephen Broughman 
Dan Kasprzyk 

Dan Kasprzyk 
Kerry Gruber 

Kerry Gruber 

Kerry Gruber 
Susan Wiley 
Dan Kasprzyk 

Dan Kasprzyk 
Kerry Gruber 

Stephen Broughman 
Marilyn Seastrom 



Third International Mathematics and Science Study (TIMSS) 

2001-01 Cross-National Variation in Educational Preparation for Adulthood: From Early Elvira Hausken 

Adolescence to Young Adulthood 

2001-05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics Patrick Gonzales 
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Title 



NCES contact 



No. 

2001- 07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

2002- 01 Legal and Ethical Issues in the Use of Video in Education Research 



Arnold Goldstein 
Patrick Gonzales 
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Listing of NCES Working Papers by Subject 



No. Title 



NCES contact 



Achievement (student) - mathematics 

2001-05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 



Adult education 

96-14 The 1995 National Household Education Survey: Reinterview Results for the Adult 
Education Component 

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96-22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 
98-03 Adult Education in the 1990s: A Report on the 1991 National Household Education 
Survey 

98-1 0 Adult Education Participation Decisions and Barriers: Review of Conceptual Frameworks 
and Empirical Studies 

1999- 1 1 Data Sources on Lifelong Learning Available from the National Center for Education 

Statistics 

2000- 1 6a Lifelong Learning NCES Task Force: Final Report Volume I 

2000-1 6b Lifelong Learning NCES Task Force: Final Report Volume II 



Adult literacy — see Literacy of adults 
American Indian - education 

1999-13 1993-94 Schools and Staffing Survey: Data File User’s Manual, Volume IV: Bureau of 

Indian Affairs (BIA) Restricted-Use Codebook 



Assessment/achievement 
95-12 Rural Education Data User’s Guide 

95-13 Assessing Students with Disabilities and Limited English Proficiency 
97-29 Can State Assessment Data be Used to Reduce State NAEP Sample Sizes? 

97-30 ACT’s NAEP Redesign Project: Assessment Design is the Key to Useful and Stable 
Assessment Results 

97-3 1 NAEP Reconfigured: An Integrated Redesign of the National Assessment of Educational 
Progress 

97-32 Innovative Solutions to Intractable Large Scale Assessment (Problem 2: Background 
Questions) 

97-37 Optimal Rating Procedures and Methodology for NAEP Open-ended Items 

97- 44 Development of a SASS 1993-94 School-Level Student Achievement Subfile: Using 

State Assessments and State NAEP, Feasibility Study 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

2001-07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 

2001-1 1 Impact of Selected Background Variables on Students’ NAEP Math Performance 
2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 

2001- 19 The Measurement of Home Background Indicators: Cognitive Laboratory Investigations 

of the Responses of Fourth and Eighth Graders to Questionnaire Items and Parental 
Assessment of the Invasiveness of These Items 

2002- 05 Early Childhood Longitudinal Study -Kindergarten Class of 1998-99 (ECLS-K), 

Psychometric Report for Kindergarten Through First Grade 



Patrick Gonzales 

Steven Kaufman 

Kathryn Chandler 

Kathryn Chandler 

Peter Stowe 

Peter Stowe 

Lisa Hudson 

Lisa Hudson 
Lisa Hudson 



Kerry Gruber 



Samuel Peng 
James Houser 
Larry Ogle 
Larry Ogle 

Larry Ogle 

Larry Ogle 

Larry Ogle 
Michael Ross 

Jeffrey Owings 



Arnold Goldstein 



Arnold Goldstein 
Arnold Goldstein 
Arnold Goldstein 



Elvira Hausken 



No. Title 

2002-06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 

Beginning students in postsecondary education 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

2001-04 Beginning Postsecondary Students Longitudinal Study: 1996-2001 (BPS: 1996/2001) 

Field Test Methodology Report 

Civic participation 

97- 25 1996 National Household Education Survey (NHES:96) Questionnaires: 

Screener/Household and Library, Parent and Family Involvement in Education and 
Civic Involvement, Youth Civic Involvement, and Adult Civic Involvement 

Climate of schools 

95-14 Empirical Evaluation of Social, Psychological, & Educational Construct Variables Used 
in NCES Surveys 

Cost of education indices 

94- 05 Cost-of-Education Differentials Across the States 

Course-taking 

95- 12 Rural Education Data User’s Guide 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 
1999-05 Procedures Guide for Transcript Studies 
1999-06 1998 Revision of the Secondary School Taxonomy 

Crime 

97- 09 Status of Data on Crime and Violence in Schools: Final Report 

Curriculum 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 

98- 09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Customer service 

1999- 10 What Users Say About Schools and Staffing Survey Publications 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2000- 04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 

1999 AAPOR Meetings 

Data quality 

97-1 3 Improving Data Quality in NCES: Database-to-Report Process 

2001- 1 1 Impact of Selected Background Variables on Students’ NAEP Math Performance 

2001-13 The Effects of Accommodations on the Assessment of LEP Students in NAEP 

2001- 19 The Measurement of Home Background Indicators: Cognitive Laboratory Investigations 

of the Responses of Fourth and Eighth Graders to Questionnaire Items and Parental 
Assessment of the Invasiveness of These Items 

2002- 06 The Measurement of Instructional Background Indicators: Cognitive Laboratory 

Investigations of the Responses of Fourth and Eighth Grade Students and Teachers to 
Questionnaire Items 



Data warehouse 

2000-04 Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meetings 



NCES contact 
Arnold Goldstein 



Aurora D’Amico 
Paula Knepper 



Kathryn Chandler 



Samuel Peng 



William J. Fowler, Jr. 



Samuel Peng 
Jeffrey Owings 



Dawn Nelson 
Dawn Nelson 



Lee Hoffman 



Sharon Bobbitt & 
John Ralph 
Jeffrey Owings 



Dan Kasprzyk 
Valena Plisko 
Dan Kasprzyk 



Susan Ahmed 
Arnold Goldstein 
Arnold Goldstein 
Arnold Goldstein 



Arnold Goldstein 



Dan Kasprzyk 
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No. 



Title 



NCES contact 



Design effects 

2000-03 Strengths and Limitations of Using SUDAAN, Stata, and WesVarPC for Computing 
Variances from NCES Data Sets 



Dropout rates, high school 

95-07 National Education Longitudinal Study of 1988: Conducting Trend Analyses HS&B and 
NELS:88 Sophomore Cohort Dropouts 



Early childhood education 

96-20 1991 National Household Education Survey (NHES:91) Questionnaires: Screener, Early 

Childhood Education, and Adult Education 

96- 22 1995 National Household Education Survey (NHES:95) Questionnaires: Screener, Early 

Childhood Program Participation, and Adult Education 

97- 24 Formulating a Design for the ECLS: A Review of Longitudinal Studies 

97- 36 Measuring the Quality of Program Environments in Head Start and Other Early Childhood 

Programs: A Review and Recommendations for Future Research 

1999- 01 A Birth Cohort Study: Conceptual and Design Considerations and Rationale 

2001-02 Measuring Father Involvement in Young Children's Lives: Recommendations for a 

Fatherhood Module for the ECLS-B 

2001-03 Measures of Socio-Emotional Development in Middle School 

2001- 06 Papers from the Early Childhood Longitudinal Studies Program: Presented at the 2001 

AERA and SRCD Meetings 

2002- 05 Early Childhood Longitudinal Study-Kindergarten Class of 1998-99 (ECLS-K), 

Psychometric Report for Kindergarten Through First Grade 

Educational attainment 

98- 1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 

Test Report 

2001- 15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test 

Methodology Report 

Educational research 

2000- 02 Coordinating NCES Surveys: Options, Issues, Challenges, and Next Steps 

2002- 01 Legal and Ethical Issues in the Use of Video in Education Research 

Eighth-graders 

2001- 05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 



Employment 

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and 
Issues 

98-1 1 Beginning Postsecondary Students Longitudinal Study First Follow-up (BPS:96-98) Field 
Test Report 

2000-1 6a Lifelong Learning NCES Task Force: Final Report Volume I 

2000- 1 6b Lifelong Learning NCES Task Force: Final Report Volume II 

2001- 01 Cross-National Variation in Educational Preparation for Adulthood: From Early 

Adolescence to Young Adulthood 



Employment - after college 

2001-15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test 
Methodology Report 



Engineering 

2000-1 1 Financial Aid Profile of Graduate Students in Science and Engineering 

Enrollment - after college 



Ralph Lee 



Jeffrey Owings 



Kathryn Chandler 

Kathryn Chandler 

Jerry West 
Jerry West 

Jerry West 
Jerry West 

Elvira Hausken 
Jerry West 



Elvira Hausken 



Aurora D’Amico 
Andrew G. Malizio 



Valena Plisko 
Patrick Gonzales 



Patrick Gonzales 



Jeffrey Owings 

Aurora D’Amico 

Lisa Hudson 
Lisa Hudson 
Elvira Hausken 



Andrew G. Malizio 



Aurora D’Amico 
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No. Title 

2001-15 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test 
Methodology Report 

Faculty - higher education 

97-26 Strategies for Improving Accuracy of Postsecondaiy Faculty Lists 
2000-01 1999 National Study of Postsecondaiy Faculty (NSOPF:99) Field Test Report 



Fathers - role in education 

2001-02 Measuring Father Involvement in Young Children's Lives: Recommendations for a 
Fatherhood Module for the ECLS-B 



Finance - elementary and secondary schools 



94-05 Cost-of-Education Differentials Across the States 

96-19 Assessment and Analysis of School-Level Expenditures 
98-01 Collection of Public School Expenditure Data: Development of a Questionnaire 

1999-07 Collection of Resource and Expenditure Data on the Schools and Staffing Survey 

1999- 16 Measuring Resources in Education: From Accounting to the Resource Cost Model 

Approach 

2000- 18 Feasibility Report: School-Level Finance Pretest, Public School District Questionnaire 



Finance - postsecondary 

97-27 Pilot Test of IPEDS Finance Survey 

2000-14 IPEDS Finance Data Comparisons Under the 1997 Financial Accounting Standards for 
Private, Not-for-Profit Institutes: A Concept Paper 



Finance - 

95- 17 

96- 16 

97- 07 

97-22 

1999- 07 

2000- 15 



private schools 

Estimates of Expenditures for Private K-12 Schools 
Strategies for Collecting Finance Data from Private Schools 
The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 
Schools: An Exploratory Analysis 

Collection of Private School Finance Data: Development of a Questionnaire 
Collection of Resource and Expenditure Data on the Schools and Staffing Survey 
Feasibility Report: School-Level Finance Pretest, Private School Questionnaire 



Geography 

98-04 Geographic Variations in Public Schools’ Costs 



Graduate students 

2000-1 1 Financial Aid Profile of Graduate Students in Science and Engineering 



Graduates of postsecondary education 

2001-1 5 Baccalaureate and Beyond Longitudinal Study: 2000/01 Follow-Up Field Test 
Methodology Report 



Imputation 

2000- 04 

2001 - 10 
2001-16 
2001-17 
2001-18 



Selected Papers on Education Surveys: Papers Presented at the 1998 and 1999 ASA and 
1999 AAPOR Meeting 

Comparison of Proc Impute and Schafer’s Multiple Imputation Software 
Imputation of Test Scores in the National Education Longitudinal Study of 1988 
A Study of Imputation Algorithms 
A Study of Variance Estimation Methods 



Inflation 

97 — 43 Measuring Inflation in Public School Costs 



Institution data 

2000-01 1999 National Study of Postsecondaiy Faculty (NSOPF:99) Field Test Report 



Instructional resources and practices 

95-1 1 Measuring Instruction, Curriculum Content, and Instructional Resources: The Status of 
Recent Work 



NCES contact 
Andrew G. Malizio 



Linda Zimbler 
Linda Zimbler 



Jerry West 



William J. Fowler, Jr. 
William J. Fowler, Jr. 
Stephen Broughman 
Stephen Broughman 
William J. Fowler, Jr. 

Stephen Broughman 



Peter Stowe 
Peter Stowe 



Stephen Broughman 
Stephen Broughman 
Stephen Broughman 

Stephen Broughman 
Stephen Broughman 
Stephen Broughman 



William J. Fowler, Jr. 



Aurora D’Amico 



Andrew G. Malizio 



Dan Kasprzyk 

Sam Peng 
Ralph Lee 
Ralph Lee 
Ralph Lee 



William J. Fowler, Jr. 



Linda Zimbler 



Sharon Bobbitt & 
John Ralph 
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No. Title 

1999-08 Measuring Classroom Instructional Processes: Using Survey and Case Study Field Test 
Results to Improve Item Construction 

International comparisons 

97-1 1 International Comparisons of Inservice Professional Development 

97-16 International Education Expenditure Comparability Study: Final Report, Volume I 

97-17 International Education Expenditure Comparability Study: Final Report, Volume II, 

Quantitative Analysis of Expenditure Comparability 
2001-01 Cross-National Variation in Educational Preparation for Adulthood: From Early 
Adolescence to Young Adulthood 

2001-07 A Comparison of the National Assessment of Educational Progress (NAEP), the Third 

International Mathematics and Science Study Repeat (TIMSS-R), and the Programme 
for International Student Assessment (PISA) 



International comparisons - math and science achievement 

2001-05 Using TIMSS to Analyze Correlates of Performance Variation in Mathematics 



Libraries 

94-07 
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Test Report 

1999- 15 Projected Postsecondary Outcomes of 1992 High School Graduates 

Postsecondary education - staff 

97-26 Strategies for Improving Accuracy of Postsecondary Faculty Lists 

2000- 01 1999 National Study of Postsecondary Faculty (NSOPF:99) Field Test Report 

Principals 

2000-10 A Research Agenda for the 1999-2000 Schools and Staffing Survey 
Private schools 

96- 16 Strategies for Collecting Finance Data from Private Schools 

97- 07 The Determinants of Per-Pupil Expenditures in Private Elementary and Secondary 

Schools: An Exploratory Analysis 

97-22 Collection of Private School Finance Data: Development of a Questionnaire 
2000-13 Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core of 
Data (CCD) 

2000-15 Feasibility Report: School-Level Finance Pretest, Private School Questionnaire 

Projections of education statistics 

1999-15 Projected Postsecondary Outcomes of 1992 High School Graduates 

Public school finance 

1999-16 Measuring Resources in Education: From Accounting to the Resource Cost Model 
Approach 



NCES contact 
Dan Kasprzyk 

Patrick Gonzales 
Arnold Goldstein 



Arnold Goldstein 
Arnold Goldstein 



Jeffrey Owings 
Kathryn Chandler 

Jerry West 
Jerry West 

Arnold Goldstein 



Peter Stowe 



Lisa Hudson 

Lisa Hudson 
Lisa Hudson 

Aurora D’Amico 
Aurora D’Amico 



Linda Zimbler 
Linda Zimbler 



Dan Kasprzyk 



Stephen Broughman 
Stephen Broughman 

Stephen Broughman 
Kerry Gruber 

Stephen Broughman 



Aurora D’Amico 



William J. Fowler, Jr. 



293 



No. Title 

2000-18 Feasibility Report: School-Level Finance Pretest, Public School District Questionnaire 



NCES contact 

Stephen Broughman 



Public schools 

97 — 43 Measuring Inflation in Public School Costs 

98- 01 Collection of Public School Expenditure Data: Development of a Questionnaire 

98-04 Geographic Variations in Public Schools’ Costs 

1999- 02 Tracking Secondary Use of the Schools and Staffing Survey Data: Preliminary Results 

2000- 12 Coverage Evaluation of the 1994-95 Public Elementary/Secondary School Universe 

Survey 

2000-13 Non-professional Staff in the Schools and Staffing Survey (SASS) and Common Core of 
Data (CCD) 

2002-02 Locale Codes 1 987 - 2000 
Public schools - secondary 

98-09 High School Curriculum Structure: Effects on Coursetaking and Achievement in 

Mathematics for High School Graduates — An Examination of Data from the National 
Education Longitudinal Study of 1988 

Reform, educational 

96-03 National Education Longitudinal Study of 1988 (NELS:88) Research Framework and 
Issues 



Response rates 

98-02 Response Variance in the 1993-94 Schools and Staffing Survey: A Reinterview Report 
School districts 

2000-10 A Research Agenda for the 1999-2000 Schools and Staffing Survey 
School districts, public 

98-07 Decennial Census School District Project Planning Report 

1999-03 Evaluation of the 1996-97 Nonfiscal Common Core of Data Surveys Data Collection, 

Processing, and Editing Cycle 

School districts, public - demographics of 

96- 04 Census Mapping Project/School District Data Book 

Schools 
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